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I.  INTRODUCTION 


A.  PURPOSE 

This  research  identifies,  evaluates,  and  selects  the  metrics  for  program 
management  (PM)  of  the  Chemical  Accountability  Management  Information  Network 
(CAMIN),  an  automated  information  system  (AIS)  that  is  in  the  Post  Deployment 
Software  Support  (PDSS)  phase. 

This  new  look  at  the  program  management  metrics  used  for  AIS  in  the  PDSS 
phase  is  required  to  ensure  that  the  system  measures  are  appropriate  to  the  phase  of  the 
system.  Metrics  provide  the  Program  Manager  (PM)  tools  to  assess  the  program  quality, 
trends,  and  requirements.  During  the  PDSS  phase,  most  systems  undergo  iterative 
development-type  activities  to  maintain  life  cycle  fimctionality.  Systems  also  require 
concurrent  maintenance  and  operations  activities  to  optimize  daily  system  operation. 

Published  software  metrics  target  assistance  in  managing  activities  specific  to  the 
software  development  phase.  These  metrics  are  applicable  during  the  PDSS  phase  to 
support  iterative  maintenance  upgrades,  but  the  metrics  used  must  also  consider  the 
issues  of  fielded  systems.  This  research  considers  typical  metrics  used  in  management  of 
software  development,  in  combination  with  typical  metrics  that  managers  can  use  in  the 
management  of  fielded  standard  systems.  The  research  focuses  specifically  on  metrics 
and  measures  for  fielded  software  systems  in  consideration  of  their  applicability  toward 
the  CAMIN. 
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B.  BACKGROUND 


This  study  investigates  the  management  and  metrics  of  the  CAMIN  system.  The 
CAMIN  system  is  a  database  for  accountability  of  chemical  weapons  (CW),  former  CW 
production  facilities  (CWPF),  and  other  data  that  merits  preservation  in  support  of  CW 
treaties.  The  system  maintains  its  inventory  controls  like  those  used  in  a  banking  system, 
requiring  transactions  that  fully  document  change  of  status,  description,  movement,  or 
destruction  of  buildings,  weapons,  or  equipment.  The  system  purpose  is  twofold:  1)  to 
enable  compliance  with  Army  wholesale  and  retail  accountability  regulations,  and  2)  to 
provide  tracking  and  reporting  required  under  the  CW  Treaty  for  short-notice  inspections, 
continuous  presence  during  destruction,  and  annual  reporting  obligations.  The  CAMIN 
supports  users  from  specific  organizations  that  have  a  CW-related  mission  within  the 
Department  of  Defense  (DoD). 

The  CAMIN  system  management  utilizes  the  typical  program  management 
metrics  of  all  programs.  The  program  measures  key  areas  like  budgeting,  funding, 
expenditures,  performance,  and  schedule.  This  study  addresses  these  common  metrics, 
and  considers  other  metrics  that  may  be  applicable  to  CAMIN. 

The  field  of  AIS  is  constantly  changing  and  expanding.  The  effective 
management  of  AIS  requires  the  use  of  specialized  management  tools  and  techniques, 
relative  to  standard  DoD  systems.  The  primary  differences  in  AIS  relate  to  work  force 
needs,  costs,  and  turnover  rates,  scheduling  and  planning  of  creative  breakthroughs,  and 
the  high  frequency  of  external  environment  changes. 
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Most  of  the  current  literature  on  managing  AIS  focuses  on  the  initial  development 
and  acquisition  of  systems.  However,  there  is  a  dearth  of  information  on  supporting  and 
maintaining  fielded  automation  systems.  Documentation  within  DoD  and  commercial 
organizations  erroneously  implies  that  the  development  effort  “completes”  the  system. 

Fielded  software  systems  require  an  accelerated  iterative  development.  A 
manager  can  adapt  to  use  developmental  software  metrics  and  measures.  However,  little 
information  exists  within  DoD  or  commercial  documents  regarding  the  other 
management  metrics  that  would  be  applicable  for  fielded  software  systems. 

The  Year  2000  (Y2K)  scare  was  indicative  of  the  need  for  more  vigilant 
management  of  fielded  software  systems.  Managers  found  systems  to  be  technologically 
backward,  and  months  of  effort  were  required  to  repair  the  flaws.  A  few  systems 
required  a  prohibitively  costly  effort  to  repair,  and  managers  had  to  replace  the  entire 
system.  For  example,  developers  are  not  available  to  correct  programs  written  in 
programming  languages  that  are  no  longer  commonly  used.  When  the  language  is  not 
common,  programmers  necessary  to  update  older  systems  are  no  longer  available  or  cost- 
effective,  and  developers  no  longer  support  the  system  languages  with  updates. 

As  managers  field  more  AIS,  the  experiences  that  are  now  collected  are  valuable 
in  managing  future  programs.  This  thesis  documents  the  valuable  information  on  metrics 
that  managers  can  apply  to  the  CAMIN  system  and  other  similar  fielded  software 
systems. 
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C.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

What  are  appropriate  metrics  and  measures  for  management  of  the  Chemical 
Accountability  Management  Information  Network? 

2.  Subsidiary  Research  Questions 

What  is  the  CAMIN  System,  and  how  does  its  management  occur? 

What  are  typical  management  metrics  and  measures  for  fielded  systems,  and  how 
do  managers  perform  management  on  fielded  systems? 

What  are  Management  Information  Systems?  What  are  the  applications  for  these 
systems? 

What  are  typical  management  metrics  and  measures  for  Development  of 
Management  Information  Systems,  and  how  do  managers  perform  management  for 
Management  Information  Systems  during  development? 

What  are  the  relevant  acquisition  phases  and  activities  for  Management 
Information  Systems?  In  what  ways  are  they  similar  to  and  different  from  those 
corresponding  to  other  types  of  systems? 

What  measures  of  effectiveness  can  help  to  assess  Management  Information 
Systems  metrics  and  measures  during  PDSS? 

How  can  the  results  of  this  research  be  generalized?  What  lessons  can  be  learned 
firom  this  analysis? 
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D.  SCOPE  OF  THESIS,  LIMITATIONS,  AND  ASSUMPTIONS 


This  thesis  addresses  metrics  for  management  of  the  CAMIN  in  the  PDSS  phase. 
It  investigates  lessons  learned  by  managers  of  the  CAMIN  and  of  other  similar  systems. 
The  thesis  includes  only  systems  in  the  PDSS  phase  that  are  managed  within  DoD. 

The  metrics  identified  and  evaluated  in  this  thesis  are  within  the  criteria  of  those 
specifically  associated  with  software  programs  and  those  associated  with  fielded  systems. 
The  common  metrics  literature  for  software  systems  are  oriented  toward  development, 
and  do  not  address  measures  that  would  specifically  apply  to  software  systems  in  use. 

This  thesis  defines  the  term  “metrics”  as  a  standard  of  measurement  and  the 
application  of  statistics  and  mathematical  analysis  to  a  specified  field  of  study.  The 
metrics  described  and  defined  in  this  thesis  are  specific  ways  of  measuring  and  evaluating 
the  defined  aspects  of  the  CAMIN  program. 

E.  METHODOLOGY 

This  thesis  methodology  employs  the  case  method.  The  case  method  includes 
three  major  elements  of  research.  First,  the  thesis  examines  system  management  for  the 
CAMIN  system.  Next,  the  thesis  presents  the  results  of  a  document  search  of  common 
metrics  that  may  be  applicable  to  CAMIN.  Finally,  the  thesis  includes  a  study  of  metrics 
for  similar  systems.  This  research  develops  and  analyzes  data  for  applicability  to  the 
CAMIN  Program. 

The  thesis  documents  the  analysis  of  these  data  and  makes  specific 
recommendations  for  the  CAMIN  program  management  metrics.  The  data  are  a 
compilation  of  common  documented  metrics  for  software  development  and  program 
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management,  together  with  management  metrics  for  other  fielded  software  systems.  This 
analysis  assesses  metrics  for  applicability  based  on  past,  current,  and  anticipated  CAMIN 
management  issues. 

F.  THESIS  ORGANIZATION 

The  first  chapter  of  this  thesis  provides  a  general  introduction  of  the  purpose, 
background,  objectives,  the  research  questions  that  apply  to  this  study,  scope, 
methodology,  and  expected  benefits  of  the  research. 

Chapter  II  provides  the  background  material  to  begin  the  research  and  analysis  of 
this  thesis.  Following  the  introductory  section  is  a  detailed  descriptive  history  of  the 
CAMIN  system,  its  missions,  and  functions,  and  how  the  organizational  structure  is  set 
up  to  operate  and  manage  the  system.  There  is  a  discussion  of  the  CAMIN  system  and  its 
status  with  regard  to  the  acquisition  process.  This  chapter  includes  significant  historic 
events  and  associated  CAMIN  program  metrics.  Finally,  Chapter  II  defines  and  discusses 
the  CAMIN  program  from  the  program  management  perspective  concerning  five  basic 
issue  areas. 

In  Chapter  III,  the  thesis  discusses  the  t5^es  of  metrics  that  managers  typically  use 
in  Management  Information  Systems.  The  chapter  presents  a  detailed  description  of 
software  system  metrics  identified  through  literature  search.  The  types  of  metrics  that 
managers  typically  use  in  program  management  of  Fielded  Systems  are  included.  This 
chapter  also  addresses  two  fielded  software  systems  that  may  be  comparable  to  CAMIN. 
This  chapter  presents  metrics  identified  through  case  study  interviews  of  other  fielded 
software  systems.  These  data  provide  metrics  and  other  data  for  analysis. 
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Chapter  FV  analyzes  the  data  presented  in  the  thesis.  Chapter  IV  discusses 
observations  from  literature  and  other  fielded  software  systems  in  reference  to  the 
CAMIN  program  areas  of  concern  that  may  warrant  metrics  and  measures.  The 
observations  create  the  basis  for  the  final  set  of  metrics  for  CAMIN.  Data  from  the  thesis 
data  collection  and  analysis  process  generate  lessons  learned  that  are  applicable  to  the 
subject. 

Chapter  V  provides  conclusions  and  recommendations.  In  addition  to  the 
conclusions  that  one  can  draw  from  the  results  and  analysis,  specific  recommendations  to 
the  CAMIN  program  and  recommendations  to  other  fielded  software  systems  Managers 
are  included.  The  final  output  of  the  study  is  a  listing  and  description  of  future  research 
topics. 

G.  EXPECTED  BENEFITS  OF  TfflS  THESIS 

The  results  of  this  study  provide  direct  benefit  to  the  CAMIN  Program  The  PM 
selected  the  metrics  historically  used  on  the  program,  but  with  thoughtful  evaluation,  the 
system  metrics  that  this  thesis  selects  can  have  a  direct  and  meaningful  effect  on  the 
program’s  cost,  schedule,  and  performance.  Actively  evaluating  and  selecting  program 
metrics  can  help  to  guide  future  planning  on  the  CAMIN  system.  The  thesis  should 
support  future  annual  planning  and  management  for  the  CAMIN  System. 

Finally,  managers  can  use  these  results  in  planning  for  the  PDSS  phase  of  future 
automation  systems.  This  study  results  in  useful  information  that  automation  system 
managers  may  find  applicable  in  planning  and  establishing  metrics  and  measures  for 
other  fielded  software  systems. 
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II.  THE  CAMIN  SYSTEM,  PAST  AND  PRESENT 


The  CAMIN  system  program  management  is  complex  due  to  the  system  size  and 
number  of  applications,  and  management  of  complex  systems  warrants  measures.  The 
program  has  a  history  of  collecting  and  interpreting  metrics.  This  chapter  provides  a 
descriptive  history  of  the  CAMIN  system  and  the  historic  approach  to  program  metrics. 
Understanding  the  history  of  the  program  is  important  to  accomplishing  the  objective  of 
this  thesis.  Every  program  is  unique,  and  requires  individual  analysis  to  determine 
appropriate  metrics  for  the  program  in  a  given  phase  of  development. 

This  chapter  describes  the  CAMIN  system,  and  its  status  with  regard  to  the 
Acquisition  Process.  The  CAMIN  Program  history  has  been  very  complex  and  unique, 
and  this  chapter  includes  a  brief  discussion  of  how  CAMIN  evolved  through  each  of  the 
acquisitions  phases. 

This  chapter  concludes  with  a  listing  of  the  key  program  management  issues  for 
CAMIN,  and  a  breakdown  of  factors  that  contribute  to  these  issues. 

A.  HISTORY  OF  CAMIN  AND  METRICS 

1.  CAMIN  Mission 

The  CAMIN  Program  has  been  in  existence  for  over  six  years.  The  CAMIN  is  a 
dual-purpose  system,  serving  the  chemical  treaties  for  the  DoD  and  the  accountability 
mission  for  the  Army’s  CW  stockpile  under  the  DoD  Single  Manager  for  Conventional 
Ammunition. 
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Key  areas  derived  from  these  two  missions  drive  the  CAMIN  configuration  and 
architecture.  The  Treaty  mission  must  support  short-notice  Treaty  inspections.  Because 
of  the  sensitive  nature  of  the  Treaty  data  and  the  potential  for  impact  on  intemational 


relations,  there  is  a  zero-tolerance  goal  in  the  CAMIN  for  data  errors  relative  to  Treaty 


reporting  and  accountability  of  chemical  materiel.  There  are  CAMIN  users  at  sites 
throughout  DoD,  and  these  users  need  to  access  real-time  data  24  hours  a  day,  seven  days 
a  week  (24-7).  System  administration  controls  permissions  to  ensure  that  only  users  with 
training  and  responsibility  can  change  the  data.  The  CAMIN  must  provide  the  ability  to 
audit  all  data  changes,  and  adequate  security  can  help  to  protect  the  data  in  CAMIN. 


CAMIN  Data 


CAMIN 

DoD  Treaty 
Accountability 

Destruction 
Reporting 


Tracking 
Tagged 
Items 

Declarations 
and  Inspections 

Locally  managed  items 


[From  Ref  15] 
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Figure  1. 


Figure  1  shows  the  types  of  data  tracked  in  CAMIN,  and  how  those  data  fit  into 
the  major  mission  areas  of  CAMIN.  As  shown  here,  data  exist  in  CAMBST  that  do  not  fall 
into  either  area,  such  as  data  for  locally  owned  stocks.  Users  choose  to  store  and 
maintain  these  data  in  CAMIN  for  convenience.  Additionally,  this  thesis  uses  the  terms 
“destruction”  and  “demilitarization”  interchangeably,  in  conjunction  with  elimination  of 
CW  assets. 

CAMIN  provides  tracking,  accountability,  and  reporting  required  for  DoD 
compliance  with  the  Chemical  Weapons  Convention  (CWC),  [Ref  18]  a  Treaty  formally 
known  as  the  “Convention  on  the  Prohibition  of  the  Development,  Production, 
Stockpiling,  and  Use  of  Chemical  Weapons  and  on  Their  Destruction.”  The  CWC  is  a 
multilateral  Treaty,  governed  from  The  Hague,  by  the  Organization  for  the  Prohibition  of 
Chemical  Weapons  (OPCW).  As  of  12  February  2001, 174  State  Parties  have  signed  the 
CWC,  and  143  countries  have  ratified  the  treaty,  including  the  United  States  [Ref.  19]. 
The  CWC  specifically  addresses  the  destruction  of  CW  and  CWPF. 

The  CAMIN  performs  all  of  the  user  requirements  using  a  familiar,  windows- 
based  interface.  Screen  captures  for  typical  representative  CAMIN  applications  are 
available  for  viewing  in  Appendix  E. 

CAMIN  tracks  locations  of  Treaty-declarable  items  within  DoD  at  the 
geographical  site,  the  declared  area  within  the  site,  the  building,  and  the  grid  within  the 
building.  The  system  tracks  item  information  for  the  CWC  such  as  nomenclature,  serial 
number,  and  tags  applied  by  inspectors.  Documentation  and  information  about  historical 
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movements,  destruction,  or  changes  in  item  status  reside  in  CAMIN.  CAMIN  maintains 
site  diagrams  for  all  declared  facilities,  as  well  as  process  flow  diagramg  of  CWPF. 
CAMIN  provides  specially  formatted  reports  for  the  short-notice  Treaty  inspections, 
while  data  from  the  system  provides  input  to  the  annual  reports  submitted  by  the  United 
States  to  the  OPCW. 

CAMIN  also  tracks  Schedule  1  chemical  material  that  the  DoD  mflintaing  for 
permitted  purposes  under  the  CWC  [Ref  18].  The  CWC  monitors  and  controls  three 
schedules  of  chemicals.  The  Schedule  1  list  mainly  comprises  the  weapons-grade  toxic 
chemicals.  Schedule  2  and  3  chemicals  include  precursors  to  toxic  chemicals,  which  are 
toxic  chemicals  that  are  commonly  used  for  industrial  purposes.  The  CAMIN  system 
does  not  currently  track  information  related  to  Schedule  2  or  3  chemicals,  as  DoD  does 
not  have  Schedule  2  or  3  chemical  assets  that  are  subject  to  CWC  monitoring.  The  CWC 
permits  State  Parties  to  develop,  produce,  otherwise  acquire,  retain,  transfer,  and  use 
toxic  chemicals  and  their  precursors  for  purposes  not  prohibited  under  the  convention. 
These  permitted  purposes  are  industrial,  agricultural,  research,  medical,  pharmaceutical, 
or  other  peaceful  purposes.  In  support  of  short-notice  inspections,  the  CAMIN  tracks 
items  that  the  DoD  retains  for  these  purposes  as  individual  items.  CAMIN  also  maintains 
the  chemical  amounts  for  Schedule  1  permitted  activities  to  ensure  that  the  DoD  remains 
in  compliance  with  Treaty  thresholds  for  maximum  storage  and  production  levels. 

In  support  of  the  CWC,  CAMIN  maintains  auditable  accountability  records  for 
CWPF  equipment  and  for  Schedule  1  permitted  items.  For  CW,  CAMIN  also  maintains 
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auditable  accountability,  not  only  for  the  CWC,  but  also  in  support  of  the  Army’s 
accoimtability  mission. 

CAMIN  maintains  strict  accountability  of  all  wholesale  CW,  chemical-peculiar 
equipment,  and  bulk  storage  of  chemical  materiel.  CAMIN  ensures  compliance  with 
U.S.  Army  and  Army  Materiel  Command  regulations  that  govern  wholesale  property 
accountability.  CAMIN  gives  a  full  audit  trail  for  change  to  status,  nomenclature,  codes, 
location,  or  destruction.  CAMIN  provides  documents  and  application  for  performing 
annual  inventories  of  the  chemical  materiel.  Data  in  CAMIN  include  building  lists,  site 
diagrams,  and  movement  and  destruction  records.  CAMIN  provides  diagrams  of  the 
buildings  and  the  specific  grid  locations  that  store  items.  The  weapons  and  production 
equipment  information  in  CAMIN  includes  not  only  location,  but  also  description,  serial 
niambers.  Army  codes  for  condition  and  ownership,  and  other  parameters. 

In  addition  to  its  two  primary  designed  purposes,  CAMIN  also  permits  system 
users  to  track  locally  owned  stocks,  defect  codes,  and  other  codes  for  munitions  that  are 
not  required  for  CWC  or  Army  wholesale  accountability. 

2.  CAMIN  Management  and  Funding  Structure 

The  Defense  Threat  Reduction  Agency  developed  the  CAMIN  system,  and  then 
transitioned  program  ownership  to  the  U.S.  Army  Soldier  and  Biological  Chemical 
Command  in  1997.  Presently,  CAMIN  program  management  resides  within  the 
Operations  Enterprise  of  the  Soldier  and  Biological  Chemical  Command  (SBCCOM). 
The  Stockpile  Management  Team  performs  functions  required  to  manage  the  CAMIN 
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program,  but  the  funding  and  approval  process  involves  multiple  organizations  within  the 
enterprise,  as  shown  in  Figure  2. 


[Developed  by  Researcher] 

Figure  2.  CAMIN  Management  Organization 

The  two  primary  sources  of  funding  for  the  CAMIN  program  are  storage  and 
treaty.  There  is  a  negotiated  agreement  for  the  distribution  of  funding  requirements, 
between  storage  and  treaty,  for  the  Operation  and  Maintenance  (O&M)  Phase  of  the 
program.  Before  transition  of  Army  wholesale  accountability  to  CAMIN,  the  Treaty 
program  funded  the  entire  system.  After  transition,  the  “customer”  who  requires  the 
system  upgrade  is  responsible  for  funding  the  upgrade.  In  essence,  the  customer  funds 
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the  effort  to  accomplish  its  specific  requirement.  For  activities  of  general  benefit,  such  as 
user  support  and  system  administration,  a  basic  allocation  shown  in  Table  1  uses  the 
number  of  active  CW  storage  sites  as  a  parameter  to  determine  funding  allocations.  The 
Treaty  portion  of  the  funding  for  these  general  costs  increases  as  CW  Stockpile  sites 
complete  destruction  activities.  In  this  table,  the  term  “CW  Sites”  refers  to  the  number  of 
active  CW  Stockpile  Storage  Sites  during  that  fiscal  year  (FY).  The  number  compares  to 
the  baseline  total  number  of  nine  stockpile  sites,  to  calculate  the  portion  of  funding  for 
Treaty  vs.  storage.  For  example,  Johnston  Atoll  Chemical  Agent  Disposal  System 
(JACADS)  completed  destruction  in  FY  01,  and  in  FY  02,  there  is  one  less  CW  site  to 
support.  These  numbers,  based  on  current  destruction  schedules,  are  subject  to  revision. 
Understanding  this  funding  and  management  structure  helps  to  understand  the  metrics 
that  this  program  must  collect.  Only  by  defining  appropriate  metrics  can  the  program 
justify  adequate  funding  to  maintain  system  operations. 
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Table  1 .  CAMIN  Funding  Allocation 
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personally  involved  with  the  issuance  of  storage  funding.  Within  the  Operations 
enterprise,  three  organizations  are  involved  in  CAMIN  management  and  funding:  1)  The 
Stockpile  Management  Team,  2)  the  Center  for  Treaty  Implementation  and  Compliance 
(CTIC),  and  3)  the  Business  Management  Team  (BMT). 

The  Stockpile  Management  Team  (SMT)  provides  the  overall  CAMIN 
management  and  serves  as  the  National  Inventory  Control  Point  and  Accountable  Officer 
for  CW  and  Chemical-Unique  Containers.  The  team  leader  participates  in  the  budgeting 
process  for  storage  funding. 

A  team  acting  as  a  program  management  office  (PMO)  performs  CAMIN 
management  functions  Avithin  the  SMT.  The  PMO  for  CAMIN  performs  business 
planning  and  budgeting  for  the  CAMIN  program.  The  majority  of  funding  for  CAMIN 
goes  into  the  support  contract,  for  user  support  and  training,  system  administration,  data 
administration,  hardware  and  software  purchase,  software  maintenance  and  support.  In 
addition  to  business  planning,  budgeting,  planning,  and  execution,  CAMIN  program 
management  functions  performed  by  Government  personnel  include  help  desk 
configuration  management,  data  management,  and  data  administration.  The  program 
areas  that  follow  provide  detailed  descriptions  of  these  tasks. 

As  stated  above,  the  National  Inventory  Control  Point  (NICP)  and  Accoimtable 
Officer  missions  for  CW  and  Chemical-Unique  Containers  missions  reside  within  the 
Stockpile  Management  Team.  In  addition  to  its  role  as  an  active  user  of  CAMIN,  the 
NICP/Accountable  Officer  also  provides  policy  guidance  for  the  CAMIN  program  in  the 
areas  of  storage  and  accountability.  Additionally,  when  the  CAMIN  program  needs 
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Dollars  in  Thousands  ($K) 


funding  to  satisfy  a  new  or  modified  requirement,  this  organization  provides  storage 
funding  approval  to  the  decision  makers  for  storage  funding  within  the  Operations 
Enterprise. 
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Figure  3.  CAMIN  Funding  History 


The  CTIC  perfonns  Treaty  Management  for  SBCCOM  and  for  the  CAMIN 
program.  They  provide  Treaty  Policy  Support  and  Treaty  Funding.  They  also  act  as 
Configuration  Control  Board  (CCB)  Co-Chair.  When  the  CAMIN  program  needs 
funding  to  satisfy  a  new  or  modified  requirement,  this  organi2ation  provides  approval  for 
Treaty  funding,  as  well  as  the  funding. 

The  BMT  witbin  die  Operations  Enterprise  manages  all  funding  that  comes  into 
the  enterprise.  The  BMT  distributes  all  funding  that  the  CTIC  later  allocates  to  the 
CAMIN  Program.  The  BMT  manages  the  storage  funding,  and  only  issues  this  funding 

to  the  CAMIN  program  with  authorization  from  die  Operations  Enteiprise  Director  and 
theNICP. 

The  chart  of  CAMIN  Funding  Histoiy  [Figure  3]  shows  the  funding  levels  and 
sources.  The  CAMIN  program  has  never  successfully  conformed  to  the  funding 
allocation  profile  that  comprised  the  agreement  between  storage  and  Treaty  in  FY  99.  As 
a  result,  managing  funding  for  the  CAMIN  program  requires  intense  coordination  every 
year.  The  PMO  must  determine  how  much  funding  is  required,  request  funding  based  on 
allocation  agreement,  and  renegotiate  based  on  actual  funding  allocated  from  the  frinding 
sources. 


3.  CAMIN  System  Description 

The  CAMIN  program  is  a  stand-alone  system,  designed  to  maintain  mission  data 
and  allow  data  retrieval  by  all  authorized  users  on  a  24-7  real-time  basis.  CAMIN 
management  strictly  controls  user  access  permissions  to  ensure  that  data  remain  as 
accurate  as  possible. 
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CAMIN  is  a  client-server  system,  where  the  CAMIN  server  is  centrally  located 
with  CAMIN  server  software,  and  users  operate  the  system  on  client  software  installed  on 
remote  workstations.  The  CAMIN  Server  is  located  at  SBCCOM,  Aberdeen,  MD. 
Workstations  are  located  at  Army  and  contractor  sites  in  the  continental  United  States 
and  the  Pacific.  The  client  platforms  for  CAMIN  use  Microsoft  Windows™  to  provide  a 
familiar  interface  for  the  user  community.  The  CAMIN  server  operates  as  a  Web  server 
to  allow  users,  which  have  proper  permission,  to  access  a  limited  set  of  CAMIN 
functionality  through  a  web  browser  interface. 

The  CAMIN  Architecture  design  depicted  in  Figure  4  provides  access  to  data 
while  maintaining  security  of  the  system  and  data.  The  primary  areas  that  can  contribute 
to  the  system  vulnerability  are  the  server,  connectivity,  and  workstation.  CAMIN  has  a 
published  Continuity  of  Operations  Plan  that  addresses  these  issues. 

The  CAMIN  Server  is  a  highly  available  cluster  design,  with  two  mirrored 
servers,  and  the  identical  data  set  resides  in  both  units.  If  the  active  server  unit  goes 
down,  the  system  automatically  switches  over  to  the  other  server  unit  in  the  cluster  and 
continues  operation.  Power  outages  at  the  server  location  are  a  source  of  concern,  and 
the  server  hardware  suite  includes  uninterruptible  power  supplies  to  smoothly  shut  down 
the  server  in  case  of  power  outage.  The  building  in  which  the  server  resides  is  vulnerable 
to  power  outages  and  air  conditioner  failures.  One  preventative  measure  against  future 
outages  is  air  conditioner  system  replacement.  The  system  administrators  back  up 
CAMIN  data  to  a  remote  location  daily  and  the  system  administrator  can  access  backup 
data  from  the  remote  location  if  there  is  a  systemic  power  failure  at  SBCCOM. 
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Precautions  can  be  justified  though  experience.  The  server  cluster  experienced  a 
systemic  and  long-term  power  failure  at  the  server  location,  caused  by  hurricane  Floyd  in 
September  1999,  while  one  of  the  client  sites  was  trying  to  prepare  for  a  short-notice 
Treaty  inspection.  The  system  administrator  used  data  backups  to  meet  the  immediate 
need,  but  plans  are  now  in  place  to  help  mitigate  future  problems. 


[After  Ref.  14] 

Figure  4.  CAMIN  System  Architecture 


Connectivity  is  another  area  of  vulnerability.  Consequently,  there  are  multiple 
ways  to  provide  access  between  CAMIN  server  and  workstation,  to  include  both  Local 
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Area  Network  (LAN)  and  phone  for  the  workstation  client  software,  and  web  access. 
The  other  issue  of  connectivity  involves  the  implementation  of  firewalls  for  security. 
Firewalls  at  server  and  client  locations  can  block  user  access  to  the  CAMIN  server. 
Firewalls  can  block  partial  or  total  connection,  depending  on  firewall  settings.  The 
CAMIN  manager  periodically  monitors  firewall  settings  and  advises  local  Corporate 
Information  Offices  (CIOs)  at  firewall  sites  to  ensure  that  CAMIN  data  and  other 
required  information  can  pass  through  the  firewall.  The  CAMIN  server  is  located  on  the 
network  within  an  area  of  the  SBCCOM  firewall  known  as  the  DMZ,  a  “demilitarized 
zone”  that  is  not  completely  within  the  firewall,  but  within  a  protected  area  at  the  firewall 
interface.  This  location  permits  users  to  access  the  system  while  retaining  well-defined 
security  benefits. 

The  workstation  and  client  software  can  also  contribute  to  CAMIN  vulnerability. 
The  CAMIN  workstations  must  have  a  special  configuration  to  operate  CAMIN  client 
software,  including  a  specific  and  atypical  operating  system,  Windows  NT  4.0.  The 
installation  of  client  software  must  be  correct  to  allow  CAMIN  functionality.  Other 
pieces  of  software  or  hardware  installed  on  the  workstation  can  interfere  with  CAMIN 
functionality.  Users  must  gain  approval  from  the  CAMIN  management  team  before 
installing  new  software  or  hardware  onto  their  CAMIN  workstation.  For  example,  one 
CAMIN  user  loaded  an  unauthorized  screen  saver  onto  her  workstation,  and  the  screen 
saver  increased  the  default  font  on  her  workstation  so  that  CAMIN  data  did  not  appear  in 
the  text  boxes  on  her  workstation.  This  change  was  very  difficult  to  diagnose  and  detect. 
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CAMIN  authorized  users  can  also  access  CAMIN  data  through  the  recently 
introduced  web  site.  Users  now  have  web  access  to  frequently  used  CAMIN  data  reports. 
An  expansion  of  web  capability  is  planned  that  allows  users  to  perform  all  workstation 
functionality  through  a  web  browser-based  interface.  This  action  resolves  all  workstation 
configuration,  cost,  and  maintenance  issues  by  eliminating  the  need  for  stringent 
workstation  requirements.  The  web  design  distills  the  firewall  connectivity  issues  to 
configuration  of  one  common  web  port. 

4.  CAMIN  Program  Areas 

CAMIN  program  management  includes  short  and  long-term  program  budgeting, 
business  and  strategic  planning,  and  organizational  and  contractual  activities.  Program 
management  also  includes  oversight  of  the  areas  of  user  support,  O&M,  and  requirements 
upgrades  for  the  CAMIN  system.  The  CAMIN  Management  Team  within  the  Stockpile 
Management  Team  is  responsible  for  all  areas  listed  in  Table  2. 

a.  Overarching  Management 

CAMIN  overarching  management  includes  budgeting  and  execution,  planning, 
reporting,  and  concept  design,  data  management,  contract  management,  configuration 
management,  and  hardware  and  software  acquisition.  The  budgeting  and  execution 
process  explanation  is  contained  in  the  previous  section. 
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Overarching 

User  Support 

Administration 

Upgrades 

Budgeting  and 
Execution 

Help  Line 

System 

Administration  and 
Security 

Design  and 
Development 

Planning,  Reporting, 
and  Concept  Design 

Training 

Data  Administration 

Code  Writing 

Data  Management 

User  Manual 

Network  and 

Firewall 

Testing 

Contracting  and 
Contract 

Management 

User  Group 

Meetings 

Web  Administration 

Fielding 

Configuration 

Management 

Newsletters  and 
Announcements 

Workstation 

Administration 

Hardware  and 

Software  Acquisition 

[Developed  by  Researcher] 


Table  2.  CAMIN  Program  Management  Areas 

Planning,  reporting,  and  concept  design  encompasses  a  large  scope  of 
activities.  Generating  strategic  and  business  plans  requires  a  study  and  application  of 
trends  in  information  technology,  and  development  and  application  of  metrics.  The  PMO 
passes  on  important  information  to  Treaty  and  accountability  management  and  through 
the  organizational  hierarchy  to  the  DoD  level.  Policies  for  the  CAMIN  program  establish 
consistency  across  users,  workstations,  and  within  SBCCOM,  Army  Materiel  Command 
(AMC),  and  Army. 
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Data  management  includes  all  work  toward  improving  the  accuracy  of 
data  in  CAMIN.  The  NICP,  system  users,  and  CTIC  are  involved  in  the  process  to 
identify  and  analyze  data  errors.  Users  or  data  administration  accomplish  data 
corrections.  Defining  the  cause  of  the  errors  leads  to  corrective  actions,  including  new 
problem/change  requests  (PCRs),  enhanced  training,  and  enhanced  user  manuals  PCRs 
document  requests  for  changes  to  the  CAMIN  software  or  system.  Users  of  CAMIN 
software  or  data  generate  the  PCRs  that  the  CCB  evaluates  for  disposition.  Data 
management  and  administration  activities  are  important,  expensive,  and  time  consuming 
to  CAMIN  Program  Management.  Challenges  to  the  CAMIN  data  come  through  user 
error,  data  migration  from  one  version  to  the  next,  and  through  policy  changes.  CAMIN 
maintains  a  strict  audit  trail  of  data  changes,  including  documentation  of  the  user  that 
caused  the  change.  Additionally,  CAMIN  uses  quality  assurance  (QA)  by  requiring  a 
second  user  login  to  “QA”  the  data  change  before  making  the  actual  data  change  in  the 
system.  Annual  inventories  and  Treaty  inspections  further  validate  the  data.  User  error 
requires  frequent  audits.  However,  proactive  measures  are  used  to  find  these  errors 
before  the  external  commimity  becomes  aware  of  them.  However,  data  migration  errors 
are  difficult  to  identify  before  they  occur.  Policy  changes  often  affect  data,  and  the 
associated  problems  are  targeted  through  analysis  and  testing. 

The  contract  management  area  ties  closely  to  budgeting,  funding 
execution,  planning,  and  upgrades.  The  CAMIN  contract  is  comprised  of  seven  phases. 
Phase  1  is  User  Support,  Phase  2  is  System  Maintenance,  Phase  3  is  Server  Maintenance, 
Phase  4  is  Meetings,  Phase  5  is  Special  Studies,  Phase  6  is  Hardware  and  Software 
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Acquisition,  Configuration,  and  Fielding,  and  Phase  7  is  System  Version  Upgrades. 
Funding  sources  provide  incremental  fimding,  and  this  makes  planning  difficult.  The  PM 
gives  the  contractor  guidelines  to  establish  which  tasks  to  devote  their  time,  and  how  to 
establish  a  workforce.  Retention  of  Information  Technology  (IT)  workers  in  the  current 
hiring  environment  requires  consistent  funding  and  interesting  and  consistent  work. 

Configuration  management  controls  software,  hardware,  data,  and 
documentation  for  the  system.  The  configuration  management  plan  defines  roles  and 
responsibilities.  The  system  configuration  manager  is  the  PM.  There  are  two  co-chairs 
of  the  CCB,  representing  Treaty  and  accoxmtability/storage  as  the  two  missions  of 
CAMIN.  The  CCB  reviews  PCRs  from  data  and  system  users,  and  has  the  authority  to 
approve  system  changes.  The  CCB  consists  of  representatives  from  funding,  users,  and 
other  primary  CAMIN  customers.  The  CCB  makes  decisions  based  on  sound  business 
criteria.  Criteria  for  analysis  include  the  need  and  benefit  of  a  change,  the  risks,  the 
potential  funding  availability,  the  costs,  and  the  difficulty/time  required  to  introduce  the 
change. 

Hardware  and  Software  Acquisition  requires  forward  planning.  The 
workstation  acquisitions  tie  to  the  three-year  maintenance  agreements  and  to  the 
fluctuating  workload  associated  with  the  destruction  effort.  Active  destruction  at  each 
major  facility  increases  the  workload  (and  workstation)  requirements  by  about  five.  The 
acquisition  of  workstation  hardware  and  software  ties  to  the  support  contract.  The 
contractor  configures  the  workstations,  loading  software  and  ensuring  functionality 
through  phone  and  LAN  connection  before  delivery.  Figure  5  shows  the  location  of 
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connection  before  delivery.  Figure  5  shows  the  location  of  CAMIN  users.  Appendix  B 
provides  a  detailed  listing  of  current  CAMIN  users  and  locations. 

b.  User  support 

User  support  covers  help  line,  training,  user  group  meetings,  web  notices, 
and  announcements.  User  competence  is  an  area  that  is  critical  to  the  CAMIN  system,  to 
help  ensure  that  CAMIN  data  is  correct  and  timely.  The  user  group  is  geographically 
diverse,  users  are  located  all  across  the  country  [Figure  5].  The  education  and  experience 
level  of  users  vary  from  SES  and  frill  Army  Colonels  to  GS-05  clerks  that  have  only  used 
a  computer  for  CAMIN.  Help  line  support  is  available  during  working  hours,  and  a  24- 
hour  emergency  pager  is  available  to  support  after-hour  emergencies.  Scheduled  training 
classes  are  offered  both  on-line  and  in  a  classroom-style  training  environment. 

Each  version  of  the  software  includes  all  relevant  docmnentation, 
including  a  full  user  manual  loaded  onto  the  CAMIN  desktop.  An  on-line  training 
database  within  the  CAMIN  system  allows  users  to  practice  an  activity  and  evaluate 
results  against  expectations.  Users  can  switch  to  the  training  database  to  test  or  practice  a 
process  before  affecting  the  live  database.  An  additional  training  tool  offered  is  the 
computer-based  training  CDs.  Annual  Data  Management  User  Group  meetings  invite  all 
users  to  participate  in  meetings  to  discuss  the  current  and  future  activities  of  the  CAMIN 
Program.  Breakout  sessions  with  user  subgroups  provide  an  additional  venue  for  contact 
and  feedback  with  iisers.  Finally,  local  administrators  at  workstation  locations  manage 
the  workstation  configuration,  including  permissions  and  access  for  users,  and  settings 
for  the  local  networks  and  firewalls  to  permit  CAMIN  access. 
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[After  Ref.  14] 

Figure  5.  CAMIN  Workstation  Sites  and  Users 


c.  Administration 

Operation  of  the  CAMIN  system  includes  system  administration  and 
security  activities,  data  administration,  network  and  firewall,  web  administration,  and 
workstation  administration.  System  administration  manages  and  maintains  server 
operations  on  a  daily  basis,  managing  file  sizes  and  system  operations.  The  system 
administrator  issues  and  maintains  user  accounts,  passwords,  and  permissions.  Security 
is  of  primary  concern.  Althou^  data  in  CAMIN  is  not  classified,  much  of  it  is  For 
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Official  Use  Only.  Passwords  and  permissions  control  system  access.  CAMIN  uses  a 
commercial  off-the-shelf  (COTS)  middleware  software  package  called  Distributed 
Computing  Environment  (DCE)  on  the  workstations  to  provide  access  controls  for  client 
security.  The  client  connections  pass  over  the  NIPRNET,  an  Army  version  of  the 
Internet.  For  the  web,  Personal  Key  Identifier  (PKI)  and  Secure  Socket  Layer  (SSL) 
maintain  a  secure  connection  and  encrypt  data  passing  over  the  web. 

Data  administration  in  CAMIN  is  required  for  correcting  data  errors, 
making  top-level  changes  to  data,  and  for  collecting  metrics  about  data  in  the  CAMIN 
system.  The  CAMIN  architecture  is  set  up  so  that  system  users  have  no  direct  access  to 
the  CAMIN  database.  The  CAMIN  system  is  extremely  complex,  with  over  800,000 
lines  of  software  code,  and  3.8  Gigabytes  of  data  files.  Table  3  summarizes  CAMIN 
lines  of  code,  and  Appendix  C  contains  information  that  is  more  detailed.  Direct  access 
to  the  CAMIN  database  to  change  system  data  is  limited  to  three  people.  The  data 
administrators  use  a  strict  control  process  to  monitor  and  approve  data  changes. 

As  stated  in  the  previous  section,  network  and  firewall  issues  can 
contribute  to  the  vulnerability  of  the  CAMIN  system.  Coordination  with  local 
administrators  helps  to  ensure  that  network  and  firewall  configurations  are  compatible 
with  CAMIN.  The  CAMIN  web  site  resides  on  the  CAMIN  server  cluster.  Web  site 
administrators  maintain  links,  post  announcements,  maintain  the  calendar,  and  ensure 
connectivity. 


28 


NT  Applications 

372,603 

Server  Applications 

344,462 

Web  Applications 

77,173 

Report  Templates 

27,000 

Database  Structure 

10,281 

System  Totals 

831,519 

[After  Ref.  4] 

Table  3.  CAMIN  Version  8.0  Lines  of  Code 

Workstation  administration  is  labor-intensive.  Due  to  the  frequency  of 
problems,  sites  assign  a  local  workstation  administrator  at  each  installation.  These 
administrators  are  not  always  well  trained  or  qualified  to  do  the  work,  but  the  individual 
provides  CAMIN  Managers  with  a  single  point  of  contact  to  send  documentation  and 
updates,  as  well  as  a  single  point  of  responsibility  for  monitoring  the  configuration  of 
workstations.  This  effort  is  necessary  because  users  and  administrators  load  unapproved 
software  and  hardware,  forget  passwords,  delete  or  overwrite  files,  dynamically  linked 
libraries  and  registiy,  and  otherwise  create  an  environment  that  prevents  CAMIN 
functionality  on  the  workstation.  DCE  facilitates  system  functionality  by  preventing 
connectivity  unless  there  is  synchronization  between  the  workstation  and  the  server. 
Lack  of  workstation  use  to  access  CAMIN  can  cause  the  clock  to  lose  synchronicity  with 
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the  server,  and  require  the  user  or  local  administrator  to  clobber  and  reconfigure  DCE 
before  using  CAMIN  applications. 

d.  Upgrades 

The  Maintenance  and  Requirements  Upgrades  include  the  areas  of  Design 
and  Development,  Code  Writing,  Testing,  and  Fielding.  These  upgrades  occur  in  the 
form  of  version  changes  or  patches.  Version  changes  have  historically  occurred  one  or 
two  times  per  year,  and  patches  may  occur  more  fi-equently.  The  Problem/Change 
Request  (PCR)  Database  keeps  track  of  all  PCRs,  classified  by  CCB  disposition, 
magnitude  of  difficulty,  ability  to  accomplish  in  a  patch,  and  whether  the  PCR  is 
associated  with  a  version  or  a  patch.  Version  Upgrade  Process:  The  process  starts  with  a 
change  being  approved  by  the  CCB,  and  with  related  requirements  for  design  or  software 
changes.  The  PM  determines  which  changes  are  needed,  and  initiates  funding  requests 
and  a  contract  modification  to  implement  the  upgrade.  Award  of  the  contract 
modification  includes  a  negotiation  of  the  exact  number  and  combination  of  PCRs,  and 
the  schedule  to  accomplish  these  system  modifications.  The  contractor  proceeds  with 
design  and  development,  code  writing,  and  then  performs  testing.  The  contractor 
provides  a  test  area  for  the  Government  to  perform  acceptance  testing.  After  Government 
acceptance,  the  contractor  publishes  and  mails  CDs,  installation  manuals,  and  Version 
Description  Documents  (VDD)  to  workstation  administrators  for  dissemination  and 
oversight  of  the  installation. 
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[Developed  by  Researcher] 

Figure  6.  CAMIN  Maintenance  Upgrade  Activities 

Patch  Process:  An  urgently  needed  change  to  the  system  typically  drives 
the  patch.  The  PM  decides  candidate  PCRs,  and  then  authorizes  the  contractor,  through 
the  Procuring  Contracting  Officer,  to  implement  the  patch.  The  contractor  and  PM  test 
the  patch,  then  make  the  patch  available  to  download  and  install  from  the  web  site.  The 
local  workstation  administrators  are  responsible  to  ensure  that  the  workstations  under 
their  purview  have  the  new  patch  installed. 

5.  CAMIN  Metrics 

The  CAMIN  PM  uses  metrics  to  assess  the  program  in  terms  of  schedtile,  budget, 
and  functionality  requirements.  The  primary  sources  for  these  metrics  are  the  contractor 
output,  PM  data  from  internal  documentation,  and  the  CAMIN  system  itself.  The  metrics 
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used  are  those  easily  accessible,  those  that  are  most  obvious,  and  those  that  are  required 
by  upper  management. 

The  support  contractor  tracks  additional  metrics  that  address  requirements, 
program  management,  quality  assurance,  configuration  management,  training,  peer 
review,  critical  resource,  maintenance,  development,  and  intergroup  coordination. 
Appendix  D  provides  a  more  detailed  list,  summarizing  the  metrics  tracked  by  the 
contractor.  Documentation  and  process  quality  are  very  important  to  the  contractor’s 
internal  procedures.  Their  activities  for  the  CAMIN  program  are  qualified  for  between 
level  3  and  4  on  the  Software  Engineering  Institute  (SEE)  Software  Capability  Maturity 
Model  (SW-CMM)  evaluation. 

The  PM  uses  metrics  that  address  areas  that  are  common  with  the  contractor.  In 
addition,  the  PM  uses  metrics  to  address  contract  management.  The  PM  generates  a 
weekly  CAMIN  report  to  measure  and  report  progress  of  the  CAMIN  data  against  Treaty 
and  destruction  goals.  The  data  in  the  weekly  report  provide  measures  of  data  and  the 
number  of  transactions  against  those  data  for  the  weekly  period.  Another  way  that 
performance  is  measured,  is  to  annually  review  the  programmatic  goals  in  the  business 
plan  against  accomplishments  for  the  year. 

The  CAMIN  program  generates  and  utilizes  metrics  summarized  in  Table  4.  The 
selection  and  application  of  appropriate  metrics  requires  organization  and  thoughtful 
consideration.  Each  metric  consumes  time  for  collection,  reporting,  and  analysis. 
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1  Metric 

Purpose 

Fxmding 
Obligated  and 
Disbursed 

Tracking  funding  helps  to  schedule  work,  to  help  predict  future 
shortcomings,  to  manage  work  force  issues,  and  to  help  assess  money 
expenditures. 

Schedule 

Tracking  schedule  helps  to  meet  user  needs  and  to  control  costs. 

User  Help 

Tracking  the  type,  duration,  and  somce  of  user  help  calls  has  benefited 
the  program  through  identifying  systemic  problems.  Evaluating  help¬ 
line  calls  can  drive  a  new  training  need  or  drive  a  change  to  the 

CAMIN  design.  Particular  users  or  user  types  may  need  additional 
training.  The  user  manual  may  need  improvement. 

Data 

Interventions 

Data  interventions  are  often  expensive,  time-consuming,  and  risky  to 
the  database.  Those  that  are  most  skilled  in  database  administration 
perform  data  interventions.  Evaluation  identifies  systemic  problems 
that  may  drive  changes  to  user  training,  user  manual,  and/or  changes  to 
the  system  software. 

Percent 

Completion 

When  developing  a  new  version  of  software,  track  the  level  of 
completion  of  each  phase. 

Training 

Training  factors  tracked  include  student  list,  types  of  training  given, 
and  feedback  reports  from  trainees.  These  measure  the  adequacy  of 
training  through  feedback  and  repeat  trainees. 

Requirements 

A  database  tracks  all  existing  requirements  for  CAMIN.  If  a 
requirement  was  completed,  the  database  retains  the  date/version  of 
completion.  This  metric  provides  a  series  of  historical  baselines,  and 
tracks  remaining  work. 

PCR 

A  database  tracks  PCRs  for  the  CCB.  This  maintains  the  list  of  the 

PCRs,  documentation  of  CCB  discussion  or  decision  on  the  action,  and 
other  related  data  elements. 
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Actions 

A  database  tracks  the  actions  from  meetings,  the  individual,  or 
organization  responsible,  and  completion  information.  This  provides  a 
measure  of  activity  that  affects  cost,  schedule,  and  performance  of 
CAMIN. 

User  List 

A  list  of  users,  their  passwords  and  permissions,  are  tracked,  and 
periodically  assessed  to  keep  the  user  list  limited  to  those  who  need 
permissions,  and  to  limit  the  permissions  to  only  those  needed.  This 
control  protects  the  CAMIN  system  and  its  data. 

Hardware/ 
Software  List 

A  list  of  program  hardware  and  installed  software  helps  to  ensure  that 
workstations  are  maintainable,  and  that  they  have  the  correct  version  of 
CAMIN  and  COTS  software. 

User  Accounts 
and  Logins 

The  CAMIN  software  has  an  application  that  provides  reports  of  user 
accounts  and  logins.  Through  this,  system  usage  and  application  usage 
is  tracked.  The  data  helps  to  determine  if  a  users  accounts  or 
permissions  need  modification.  The  data  can  help  determine  if 
applications  are  under  or  over  utilized. 

[Developed  by  Researcher] 

Table  4.  CAMIN  Program  Management  Metrics 


B.  CAMIN  AND  THE  ACQUISITION  PROCESS 

CAMIN  is  in  the  Operations  and  Support  Phase  of  the  program.  For  automation 
systems,  this  phase  corresponds  to  fielded  software  systems.  Like  typical  software 
systems,  the  CAMIN  goes  through  an  iterative  process  of  development,  production,  and 
deployment  of  system  maintenance  upgrades  and  enhancements.  This  process  of  cyclical 
changes  to  the  system  is  often  confusing  to  anyone  that  is  not  familiar  with  software 
systems. 

The  DoD  Acquisition  Model  (Figure  7)  shows  the  basic  phases  of  the  acquisition 
process.  This  section  provides  a  brief  description  of  the  history  of  the  CAMIN  Program 
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and  significant  events  within  the  acquisition  process  framework.  It  is  important  to  study 
the  history  of  CAMIN  versions  in  order  to  understand  that  CAMIN  development  has 
continued  well  into  the  PDSS  phase.  Each  version  update  has  added  to  the  fimctionality 
of  CAMIN  to  meet  the  original  functional  requirements,  while  making  incremental 
enhancements  to  the  functionality  based  on  user  needs.  Version  upgrades  continue  to  be 
required  through  the  life  of  the  system. 

1.  Concepts  and  Technology  Development 

The  developers  spent  little  time  on  the  Concept  and  Technology  Development  for 
CAMIN.  In  1994-95,  the  DoD  created  an  urgent  requirement  to  develop  a  system  for 
tracking,  verifying,  and  reporting  Chemical  Treaty  required  data.  The  basis  of  this  urgent 
requirement  was  the  imcertainty  of  when  the  United  States  Senate  would  ratify  the  first  of 
the  pending  chemical  treaties.  While  common,  urgency  is  never  a  good  environment  to 
develop  software,  and  this,  ultimately,  caused  long-term  problems. 

The  CAMIN  developer,  the  Defense  Threat  Reduction  Agency,  then  Defense 
Nuclear  Agency,  had  an  existing  contract  with  TRW,  then  BDM  Federal,  to  develop  the 
Compliance  Management  and  Tracking  System  (CMTS)  for  all  DoD  Treaty  data. 

The  CAMIN  design  uses  the  experience  of  two  prior  existing  systems.  The 
CAMIN  system  was  initially  an  add-on  to  CMTS.  Therefore,  developers  used  CMTS  as 
the  template  for  system  architecture  and  initial  design  of  CAMIN.  The  other  system 
involved  in  the  development  process  was  the  existing  Accoimtable  system,  the  Toxic 
Chemical  Munitions  (TCM)  Database.  The  TCM  was  insufficient  to  meet  Treaty  needs, 
and  was  not  robust  enough  to  continue  operation  as  destruction  began  and  transactions 
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increased  significantly.  The  TCM  design  was  very  limiting.  The  language  was  dBase 
III,  and  it  used  a  series  of  stand-alone  workstations.  TCM  users  transferred  local 
accoimtability  data  to  the  Accountable  officer  periodically  over  secure  phone  lines. 
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When  the  developers  began  working  to  merge  the  architecture  of  CMTS  with  the 
fimctionality  of  TCM,  the  results  became  very  complex.  Due  to  complexity  and  unique 
accountability  requirements,  the  system  concept  eventually  grew  into  a  system  that  is 
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completely  separate  from  CMTS.  CAMIN  was  now  ready  to  enter  the  System 
Development  and  Demonstration  phase. 


[From  Ref.  14] 

Figure  8.  CAMIN  Program  History 

2.  System  Development  and  Demonstration 

The  System  Development  and  Demonstration  of  the  CAMIN  included  basic 
software  development,  with  user  involvement.  The  developers  held  a  series  of  User 
Group  meetings  to  discuss  concepts  and  ideas.  The  developers  created  a  proposed 
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architecture  and  presented  it  to  users.  They  also  presented  sample  screens  and  views  to 
show  the  users  how  the  system  would  look. 

The  urgency  for  Treaty  needs  led  to  an  interim  goal  for  CAMIN  to  provide  only 
minimal  functionality.  In  this  first  version,  applications  had  no  inherent  process  other 
than  to  convert  inputted  data  into  the  required  format.  As  time  passed  and  managers 
grew  more  frantic,  the  Treaty  hierarchy  decided  to  release  a  first  version  that  addressed 
only  the  CW  and  accountability.  This  decision  resulted  in  the  delay  in  full  meeting 
requirements  for  CWPF  and  Permitted  Activities  applications  until  later  versions. 

The  development  of  CAMIN  was  truncated,  and  Version  I.O  fielding  occurred  at 
three  beta  sites.  Version  2.0  contained  additional  requirements  and  incorporated  user 
feedback  fi-om  Version  1.0.  The  transition  from  R«feD  funding  to  primarily  O&M 
funding  occurred  in  this  timefi-ame. 

3.  Production  and  Deployment 

Developers  fielded  Version  2.0  to  all  user  sites.  In  versions  3.0  and  4.0,  the 
developer  included  additional  functionality.  With  the  new  functionality,  the  system  came 
closer  to  meeting  the  original  system  requirements.  With  version  5.0,  the  transition 
process  began  with  the  movement  of  contract  management  activities  from  the  Defense 
Threat  Reduction  Agency  (DTRA),  as  the  Research  and  Development  agency,  to 
SBCCOM  for  Operations  and  Maintenance.  The  transition  process  included  a  full 
Independent  Verification  and  Validation  Test  (IV&V),  using  an  external  agency  to  design 
and  perform  the  test.  The  schedule  in  Figure  8  shows  the  transition  period,  spanning 
fi-om  FY  98  through  FY  99.  During  transition,  SBCCOM  awarded  a  sole  source  contract 
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to  TRW  for  O&M  of  CAMIN.  The  developer  transitioned  the  system  in  December  1998, 
Avith  a  couple  of  outstanding  actions.  Development  of  Computer-Based  Training  (CBT) 
CDs  by  DTRA  enabled  multimedia  system  training.  In  addition,  DTRA  would  also  fund 
the  required  testing  to  accomplish  Y2K  requirements  imposed  by  DoD.  Once  CAMIN 
replaced  TCM  to  become  the  Army  accountable  system  in  1998,  storage  funding  became 
part  of  the  overall  funding  for  the  management  of  CAMIN. 

4.  Operations  and  Support 

The  Operations  and  Support  phase  of  CAMIN  encompasses  a  wide  variety  of 
program  management  activities,  as  described  in  this  document.  In  addition,  new  software 
versions  continue  to  meet  the  changing  needs  of  the  users,  the  hardware/software/network 
environment,  and  the  DoD.  System  and  data  users  continue  to  identify  new  and  refined 
requirements  for  CAMIN  functionality.  Changes  included  new  data  output  or  input 
formats.  These  types  of  changes  to  requirements  can  affect  the  structure  of  the  database 
and  cause  significant  revisions  to  the  system.  The  environment  changes  also  drive 
CAMIN  changes.  Firewalls  and  other  network  issues  can  require  firequent  system 
revisions.  It  is  not  practical  to  adopt  new  software  versions  as  soon  as  the  industry  makes 
them  available,  but  the  CAMIN  system  must  eventually  migrate  to  receive  the  support 
,and  to  remain  compatible  with  the  external  environment.  The  DoD  has  recently  imposed 
significant  security-related  policies  that  affect  the  CAMIN  workstations,  server,  and  web 
site,  as  well. 

The  CAMIN  Version  8.0  fielding  occurred  in  FYOl.  A  significant  list  of 
approved  PCRs  is  ready  to  incorporate  into  the  next  version  of  CAMIN.  Once  funded. 
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the  program  to  reduce  the  dependence  on  CAMIN  workstations  by  developing  and 
providing  web-enabled  applications  for  data  entry  and  data  access  users  can  begin. 

C.  AREAS  OF  INTEREST  FOR  CAMIN  PROGRAM  MANAGEMENT 

To  summarize  this  chapter,  the  major  issue  areas  of  CAMIN  program 
management  provide  a  basis  for  analysis  of  metrics.  The  issues  listed  in  this  section  are 
interrelated,  but  each  is  a  distinct  area  for  program  management  concern.  It  is  important 
to  view  program  management  metrics  in  terms  of  program  issues.  The  selected  metrics 
must  address  all  of  these  key  areas. 

1.  Funding 

As  previously  discussed,  funding  levels  for  the  CAMIN  Program  have  been 
unpredictable,  but  have  reduced  over  the  last  two  years.  The  funding  deficit  and 
uncertainty  cause  significant  problems  for  the  CAMIN  system.  Improved 
commumcation  and  controls  mitigates  the  funding  problems.  However,  measures  of 
funding  are  needed  to  address  the  following  areas:  understanding  the  funding  needs  based 
on  program  management  requirements,  communication  of  fimding  justification  to  fund 
managers,  and  funding  availability  to  meet  program  requirements. 

2.  System  Availability 

System  availability  is  critical  for  two  key  reasons.  First,  users  must  be  able  to 
access  and  update  CAMIN  data  in  support  of  short-notice  Treaty  inspections. 
Availability  is  critical  to  enable  users  to  accomplish  timely  input  of  inventory 
transactions.  Availability  is  dependent  on  connections,  configuration,  and  power.  To 
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meet  PM  requirements  for  system  availability,  areas  that  need  to  be  addressed  are:  system 
software  capability  to  perform  customer  requirements,  power  to  the  system,  connectivity 
through  the  network,  firewall,  and  other  security  systems,  workstation  configuration  and 
user  account  maintenance,  and  system  maintenance  to  keep  the  system  running 
efficiently. 


3.  Data  and  Output  Accuracy 

The  critical  missions  of  Treaty  and  accountability  drive  the  need  for  data 
accuracy.  Treaty  requires  accuracy  to  help  maintain  U.S.  integrity  with  the  international 
inspectors.  The  accountability  mission  requires  a  system  with  substantial  data  protections 
and  an  audit  trail,  with  the  aim  of  achieving  100%  accuracy.  Accuracy  relates  to  the 
system  design,  to  the  user  permissions,  and  to  the  competency  of  system  users.  An 
oversight  organization  also  needs  to  periodically  audit  the  data.  In  this  case,  both  Treaty 
and  accountability  should  audit  data  for  their  respective  requirements.  Factors  in  this 
process  are  listed  here;  training  of  users,  help  line  for  users  of  the  system  and  data,  user 
manuals,  ease  of  use  and  data  protections  in  CAMIN  software,  data  audit  by  cognizant 
organizations,  and  data  interventions  to  correct  user  errors  and  to  perform  mass  changes 
to  system  data. 


4.  Experienced  IT  Support 

Retention  of  IT  support  staff  is  important  to  reduce  the  extended  learning  curves 
associated  with  working  on  complex  systems.  The  program’s  ability  to  retain  developers 
and  maintainers  of  CAMIN  is  dependent  on  consistent  funding,  consistent  work,  and 
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overcoming  technological  challenges.  The  CAMIN  program  has  sufficient  work 
requirements  if  funding  were  available.  Consistent  workload  through  iterative 
maintenance  activities  can  help  to  retain  an  experienced  workforce.  However, 
maintaining  the  consistent  workforce  involves  leveling  of  tasks  to  achieve  a  consistent 
level  of  work,  challenging  the  staff  by  using  current  or  leading  technology,  and  using  a 
contract  vehicle  that  provides  adequate  funding  and  work  levels. 

5.  Requirements  Changes 

The  area  of  requirement  definition  and  associated  changes  is  critical  to  continued 
CAMIN  functionality  and  user  satisfaction.  Through  the  history  of  CAMIN,  these  factors 
have  driven  the  program  changes.  The  hardware  and  software  require  periodic  upgrade 
to  maintain  continued  operation  and  maintenance.  The  users  and  customers  have  added 
requirements  and  enhancements  each  year  to  reduce  processing  time,  reduce  errors,  and 
improve  the  system.  In  order  to  assess  revisions  to  system  requirements,  the  following 
areas  must  be  addressed:  changes  to  COTS  and  hardware,  network  changes,  state  of  the 
art  technology  and  architecture,  and  evaluating  changes  in  terms  of  time  to  accomplish 
change,  and  the  priority  and  complexity  of  these  changes. 
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III.  DATA  PRESENTATION:  CANDIDATE  METRICS 


Metrics  have  become  commonplace  in  program  management,  and  metrics  tools 
are  available  through  literature  and  through  case  analysis.  In  this  chapter,  data  collection 
and  analysis  provides  a  thorough  list  of  the  types  of  metrics  that  may  be  appropriate  for 
CAMIN  and  other  systems  in  the  PDSS  phase.  This  chapter  also  contains  a  discussion  of 
the  types  of  metrics  that  managers  may  use  in  Management  Information  Systems  in 
general.  The  term  “Management  Information  Systems”  refers  to  systems  designed  to 
automate  business  processes,  rather  than  software  systems  embedded  in  weapon  systems, 
vehicles,  and  the  like. 

The  metrics  for  management  of  software  that  exist  in  the  current  literature  are 
oriented  toward  the  development  of  software,  rather  than  sustainment  of  deployed 
systems.  When  the  CAMIN  system  undergoes  its  iterative  software  changes,  the 
development-related  metrics  remain  valid.  However,  CAMIN  is  a  deployed  system,  and 
may  warrant  measurement  in  terms  of  customer  satisfaction,  user  accessibility,  including 
readiness  and  downtime,  help  desk  support,  and  other  feedback  tools.  Therefore,  Chapter 
in  also  discusses  of  the  types  of  metrics  that  Program  Managers  typically  use  in  the 
management  of  fielded  systems. 

The  discussion  of  metrics  in  this  chapter  considers  both  DoD  system  metrics  and 
commercial  metrics.  Use  of  commercial  best  practices  is  a  tenet  of  the  DoD  5000 
guidance  on  acquisition  reform  within  DoD,  and  so  consideration  of  Government  and 
commercial  practices  should  help  to  arrive  at  the  optimal  metrics  selection.  The  chapter 
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also  examines  the  metrics  of  other  fielded  software  systems  that  are  comparable  to 
CAMIN,  based  on  the  life  cycle,  fimding,  management  style,  and  type  of  system. 

A.  SOFTWARE  METRICS 

The  Clinger-Cohen  Act,  otherwise  known  as  the  Information  Technology 
Management  Reform  Act  of  1996  [Ref  5],  provides  tools  for  management  of  IT  within 
the  U.S.  Government.  These  tools  help  to  manage  funds  and  contracts  to  acquire  IT  more 
efficiently  by  providing  flexibility  in  the  acquisition  process.  The  Clinger-Cohen  Act 
requires  the  use  of  measures  for  software  acquisition  and  management  specifically,  as 
follows: 

The  head  of  an  executive  agency  shall  (3)  ensure  that  performance 
measures  are  prescribed  for  information  technology  used  by  or  to  be 
acquired  for,  the  executive  agency  and  that  the  performance  measurements 
measure  how  well  the  information  technology  supports  programs  of  the 
executive  agency. 

[From  Ref  5,  Section  5123(3)] 

To  establish  valid  metrics  for  software  development  and  production,  managers 
need  tools  to  establish  overall  process  quality.  PMs  can  require  that  software  developers 
certify  the  quality  of  the  software  production  process  through  standards  established  in 
ISO  9000  and  the  SEI  Software  Capability  Maturity  Model  (SW-CMM)  [Ref  20].  Yet, 
software  performance  parameters  are  difficult  to  define  and  measure,  as  the  definition  of 
parameters  for  a  system  changes  based  on  platform,  programming  language,  and  user 
interface.  Additional  factors  in  software  measurement  include  the  fast-changing 
environment  in  which  the  software  must  remain  functional.  The  changes  in  the  platform 
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and  networks  are  fluid.  Requirement  creep  is  also  more  common  with  software-intensive 
systems  than  with  strictly-hardware  systems.  In  addition,  integration  testing  often 
requires  completed  software  and  hardware.  The  metrics  associated  with  successful 
systems  development  projects  in  successful  companies  are  those  that  deliver  on  schedule, 
relatively  close  to  budget,  and  with  high  levels  of  quality  and  reliability. 

1.  Management  Information  Systems 

This  thesis  makes  a  differentiation  between  Management  Information  Systems 
and  other  systems.  Systems  compared  here  have  more  similarities  than  differences, 
especially  when  analyzed  within  the  DoD  acquisition  framework.  However,  the 
differences  have  an  effect  on  decisions  about  how  to  plan  for  and  to  manage  these 
systems.  Management  Information  Systems  require  frequent  changes  due  to  dependence 
on  platform  and  COTS.  Table  5  shows  a  comparison  of  the  key  program  management 
parameters  for  each  of  the  three  program  types.  These  differences  are  generalizations, 
but  are  typically  true. 

Table  5  shows  that  requirements  are  very  unstable  for  software  systems,  relative 
to  hardware.  External  sources,  such  as  environment  changes,  user  needs,  COTS  changes, 
and  platform  changes  drive  modification  to  the  system  requirements.  Development  is 
iterative  for  software-based  systems.  Complex  systems  may  experience  iterative 
development  for  many  years.  Additionally,  development-like  activities  occur  as  part  of 
the  process  to  perform  maintenance  upgrades  to  the  system  to  meet  changing  system 
requirements. 
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Management  Embedded  Software  Standard  Hardware 
Information  Systems  Systems  Systems 


Requirements 

Changes  through  the 
system  life,  driven  by 
platform, 

environment,  users 

Changes  through  the 
system  life,  driven  by 
users  and  interface 
(more  stable  platform 
and  environment) 

Theoretically  frozen 
at  ORD,  really  frozen 
at  materiel  release 

Development 

Iterative,  costly, 
through  the  life  of  the 
program,  dependent 
on  experts  and 
creativity 

Somewhat  iterative, 
but  limited  by 
hardware  platform 

One  time-  Waterfall 
to  Production 

Testing 

Beta  and  IV&V 

DT  and  OT 

DT  and  OT 

Production 

Print  CDs 

Install  in  production 
hardware 

Production  may  occur 
over  multiple  years 

Fielding 

Mail  CDs  and 
instructions,  or  make 
available  over  Web 
site 

Concurrent  with 
fielding  of  platform 

Fielding  may  occur 
over  multiple  years 

Technical 

and 

Engineering 

Data 

Source  Code  with 
comments.  Database 
Dictionary 

Source  Code  with 
comments.  Database 
Dictionary,  interface 
documents 

Drawings  and 
Specifications 

Training  and 
Manuals 

Required  and  updated 
about  annually  with 
each  version 

Required 

Required 

Operations 

and 

Maintenance 

Intense,  with 
revisions  required 
about  annually 

Concurrent  with 
periodic  platform 
maintenance 

Periodic 

PDeveloped  by  Researcher] 

Table  5.  Comparing  Management  Information  Systems  to  Other  Systems 
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Testing  for  Management  Information  Systems  can  often  take  place  in  an  office 
environment.  Through  beta  testing,  users  provide  feedback  on  the  initial  system 
functionality  and  operations. 

Production  for  software  is  simple.  Once  the  version  testing  is  complete, 
production  may  include  printing  Compact  Disks  (CDs)  with  the  installation  code  loaded. 
Changes  to  Management  Information  Systems  are  relatively  easy  to  field.  Fielding  often 
involves  the  relatively  simple  process  of  printing  and  mailing  CDs  with  installation 
instructions.  Unlike  hardware,  software  changes  are  easy  to  code  and  field,  as  well.  The 
limitation  on  system  upgrades  comes  primarily  from  the  areas  of  testing  and 
documentation.  These  processes  are  more  time-consuming  and  costly  for  software  than 
for  hardware.  Additionally,  the  technical  and  engineering  data  are  different  for  software 
and  hardware. 

For  Management  Information  Systems,  there  is  a  greater  flexibility  in  the  areas  of 
training  and  manual  generation,  revision,  and  support.  Training  can  take  place  on  a 
computer  system  with  only  a  simple  interface,  and  for  embedded  software  or  hardware, 
modeling  and  simulation  can  provide  a  realistic  training  environment.  However,  training 
is  not  complete  without  spending  time  on  the  actual  hardware  system. 

The  majority  of  activities  involved  in  the  O&M  of  software  requires  technical 
expertise,  experience,  and  directed  knowledge  of  the  system  architecture  and  code.  Data 
administration  and  software  changes  introduce  the  most  risk  when  performed  by  less 
skilled  personnel.  There  are  several  levels  to  perform  maintenance  of  most  hardware 
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systems,  and  the  maintainers  can  more  readily  understand  the  system  through  technical 
drawings  and  other  documentation. 

2.  Department  of  Defense  Requirements  for  Software  Metrics 

The  DoD  and  its  Services  have  been  in  the  forefront  of  automation  and  software 
development  since  the  first  computer  was  developed.  For  example,  the  DoD  sponsored 
the  early  development  of  supercomputers  to  calculate  ballistics,  the  computer  language 
ADA,  and  the  Oracle  database.  In  1992,  DoD  recommended  the  four  Basic  Measures  for 
software  systems  shown  in  Table  6. 


Unit  Of  Measure 

Characteristics  Addressed 

Counts  of  physical  source  lines  of  code 

Size,  progress,  reuse 

Counts  of  staff  hours  expended 

Effort,  cost,  resource,  allocations 

Calendar  dates 

Schedule 

Counts  of  software  problems  and  defects 

Quality,  readiness  for  delivery,  improvement 
trends 

[After  Ref  2] 

Table  6.  DoD  Recommended  Measures  from  1992 


The  DoD  requirements  for  software  measures  established  in  this  1992  document 
did  not  change  significantly  when  addressed  again  in  1996.  In  a  1996  policy  memo  [Ref. 
17],  OSD  requires  that,  at  a  minimum,  software  metrics  should  address  the  following  six 
areas: 
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1 .  Schedule  and  progress  regarding  work  completion, 

2.  Growth  and  stability  regarding  delivery  of  the  required  capability, 

3.  Funding  and  personnel  resources  regarding  the  work  to  be 
performed, 

4.  Product  quality  regarding  the  delivered  products, 

5.  Software  development  performance  regarding  the  capabilities  to 
meet  program  needs,  and 

6.  Technical  adequacy  regarding  software  reuse,  ADA,  and  use  of 
standard  data  elements 

The  U.S.  Army  has  developed  a  program  called  the  Software  Test  and  Evaluation 
Panel  (STEP)  to  address  software  metrics  and  their  use.  DA  Pamphlet  73-7,  dated  25 
July  1997  [Ref.  10]  provides  the  published  list  of  Army  Software  Metrics.  Table  7 
summarizes  the  list  of  metrics  included  in  the  document.  The  Air  Force  also 
recommends  the  Army  STEP  Metrics  for  program  management  of  software  [Ref.  9]. 

The  STEP  Metrics  use  many  standard  software  metrics,  adopting  best  commercial 
practices.  These  metrics  are  inclusive  of  fielded  systems,  including  measures  of  mean 
down  times,  fault  profiles,  and  reliability.  The  three  categories  for  metrics  are 
management,  requirements,  and  quality. 
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QUALITY  CATEGORY  REQUIREMENTS  MANAGEMENT 

_  CATEGORY  CATEGORY 


METRICS 


OBJECTIVES 


MEASUREMENTS 


Cost 

Schedule 

Computer  resource 
utilization 

Software  engineering 
enviromnent 

Track  software 
expenditures 

Track  schedule  adherence 

Track  planned  and  actual 
resource  use 

Quantify  developer 
software  engineering 
environment  maturity 

Dollars  spent  vs.  dollars 
allocated 

Milestone/event  slippage 

Percent  resource  capacity 
utilized 

Computed  maturity  level 

Requirements  traceability 
Requirements  stability 

Track  requirements  down 
to  code 

Track  changes  to 
requirements 

Percent  requirements 
traced 

Number  of  requirements 
changes 

Design  stability 
Complexity 

Breadth  of  testing 

Depth  of  testing 

Fault  profiles 

Reliability 

Track  design  changes 
Assess  code  quality 

Track  testing  of 
reqmrements 

Track  testing  of  code 

Track  open  vs.  closed 
anomalies 

Assess  software  mission 
failures 

Measure  down  time 

Stability  index 

Complexity  indices 

Percent  requirements 
tested 

Percent  requirements 
passed 

Degree  of  code  testing 

Number  and  types  of 
faults, 

average  open  age 

Mean  time  between 
failmes 

Restoration  times 

[From  Ref.  16] 

Table  7.  Army  Software  Test  and  Evaluation  Panel  Metrics 
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Management  metrics  address  the  basic  areas  of  cost  and  schedule.  Further,  the 
maturity  level  is  an  assessment  of  the  processes  used  by  the  software  organization,  in 
accordance  with  the  SEI  SW-CMM.  Within  the  requirements  category,  traceability  and 
stability  establish  quantitative  values  for  the  requirements  generation  and  change 
processes.  The  quality  of  the  system  provides  a  series  of  measures  that  help  managers  to 
assess  how  well  the  system  ftmctions  through  changes,  complexity,  testing,  and 
measurement  of  failures. 

3.  Commercial  Software  Metrics 

Research  shows  that  published  metrics  for  software  systems  support  the 
development  phase,  without  specifically  addressing  support  of  fielded  systems. 
Commercial  software  metrics  do  not  introduce  additional  metrics  beyond  those  used  for 
DoD.  Therefore,  the  set  of  software  metrics  established  for  DoD  and  the  Army  is 
inclusive  of  the  typical  measures  and  tools  that  commercial  industry  uses  for  management 
of  software  programs. 

B.  FIELDED  SYSTEM  METRICS 

Fielded  systems  measurement  should  address  issues  related  to  continuous 
management  of  the  system,  and  issues  relating  to  the  use  of  the  system.  These  metrics 
include  those  that  come  directly  from  the  system  customers.  The  list  of  customers 
includes  upper  management,  those  involved  in  funding,  and  those  who  perform  data 
input,  those  who  access  system  data,  and  the  system  maintainers  and  administrators. 
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Aspects  of  use  that  merit  consideration  for  metrics  are  user  satisfaction,  response  time, 
failure  rates,  equipment,  spares  and  replacements,  and  quality  problems. 

The  measures  of  fielded  systems  should  provide  specific  outcomes.  Cost  drivers 
related  to  system  operations  are  key  areas  that  warrant  measurement.  Cost  drivers 
include  user  help  and  training,  maintenance  of  software,  (e.g.,  correction  of  defects, 
enhancements),  hardware  and  software  purchases,  and  data  maintenance  and 
administration.  These  cost  drivers  can  help  to  develop  metrics  that  continuously 
diagnose  problems  and  assess  the  outcomes  of  corrective  actions. 

1.  Department  of  Defense  Logistics  Metrics 

The  DoD  establishes  metrics  for  its  fielded  systems  based  on  established  criteria. 
The  basic  goals  with  fielded  systems  are  to  maintain  system  readiness  through 
maximizing  availability  and  minimizing  maintenance  factors. 

The  DoD  environment  differs  significantly  from  the  commercial  enviromnent, 
because  systems  are  in  use  for  much  longer  in  the  military  environment  than  in  the 
commercial  environment.  These  aging  systems  require  more  frequent  maintenance 
activities.  Maintenance  on  these  older  systems  is  more  difficult,  requiring  better  records 
and  repair  parts.  The  short  life  of  commercial  automation  hardware  and  software,  and  the 
associated  support  cycle  are  not  consistent  with  the  military  concept. 

Table  8  provides  a  list  of  the  Objectives  and  Performance  Measures  that  are 
included  in  the  DoD  FY  2000  Logistic  Strategic  Plan  [Ref  7].  These  measures  are  part 
of  the  overall  logistic  plan  for  all  fielded  systems  within  DoD.  Readiness  is  a  primary 
concern  for  logistics,  and  most  of  these  DoD  measures  relate  to  readiness.  In  general,  the 
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DoD  is  concerned  with  having  systems  that  are  available  and  perform  well,  while 
controlling  costs. 

The  objective  to  improve  strategic  mobility  to  meet  Warfighter  requirements  is  to 
help  achieve  readiness  from  mobility  of  assets.  Mobility  is  only  an  issue  for 
Management  Information  Systems  in  the  rare  case  where  a  site  needs  a  node, 
workstation,  and  connectivity,  urgently.  Having  assets  available  to  address  these  urgent 
requirements  can  satisfy  another  objective,  to  fully  implement  joint  Total  Asset  Visibility 
(TAV)  across  DoD.  Knowing  the  location,  configuration,  and  usage  of  workstations  and 
user  accounts  is  the  key  to  meeting  this  objective  for  Management  Information  Systems. 


1  OBJECTIVE 

MEASURES 

Optimize  support 
to  the  Warfighter 

Measure:  Attain  the  specified  percentage  of  Level  “A”  weapon 
systems  meeting  their  targeted  Mission  Capable  (MC)  Rate  through 

FY  2006.  Develop  a  documented  baseline  of  applicable  Military 
Service  MC  rates  by  the  end  of  FY  2001 .  Establish  target  MC  rates 
for  the  end  of  FY  2006,  and  track  progress  in  attaining  these  targets 
(i.e.,  the  percentage  of  increased/decreased  MC  rates)  annually 
beginning  at  the  end  of  FY  2001.  The  Military  Services  will  develop 
the  capability  for  quantifying  the  actual  and  target  MC  rates  based  on 
individual  weapon  systems,  weapon  system  categories  or  Service 
composite,  as  appropriate,  to  provide  meaningful  performance 
information.  The  Defense  Logistics  Agency  will  develop  the 
capability  for  reporting  its  Customer  Wait  Time  (CWT)  baseline  and 
annual  progress  toward  the  Defense  Logistics  Agency  FY  2006  CWT 
target,  by  Service  or  consistent  with  the  Services’  weapon  system 
categories.  Each  Component  will  report  its  composite  progress 
against  its  target(s)  annually,  beginning  at  the  end  of  FY  2001 . 

Improve  strategic 
mobility  to  meet 
Warfighter 
requirements 

Measme  1 :  By  the  end  of  FY  2006,  achieve  a  cargo  airlift  capacity 
and  sealift  surge  and  afloat  pre-position  capacity  to  meet  the  validated 
requirements  in  the  current  Mobility  Requirements  Study. 

Measure  2:  Develop  a  measurement  plan  and  goals  for  mobility 
infrastructure  and  mobility  process  improvements  by  the  end  of  FY 

2001 .  Achieve  those  goals  by  the  end  of  FY  2006. 
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Implement 
Customer  Wait 
Time  (CWT)  as 
the  DoD  logistics 
metric 

Measure  1 :  Develop  the  process  for  definition  and  measurement  of 
Customer  Wait  Time  (CWT)  by  the  end  of  FY  2001. 

Measure  2:  Fully  implement  CWT  measurement  for  100  percent  of 
all  selected  segments  by  the  end  of  FY  2006. 

Fully  implement 
joint  Total  Asset 
Visibility  (TAV) 
across  DoD 

Measure:  Determine  user/business  methods,  asset  information 
requirements  and  associated  measures  by  the  end  of  FY  2000, 
implement  100  percent  of  requirements  by  the  end  of  FY  2006. 

Reengineer/ 

modernize 

applicable 

logistics 

processes/ 

systems 

Measure:  Develop  Component  logistics  processes/systems 
modernization  plans  by  the  end  of  FY  2001,  and  increase  the 
proportion  of  modernized  logistics  business  systems  according  to 
those  plans  by  the  end  of  FY  2006.  The  Military  Services,  the 

Defense  Logistics  Agency,  and  the  U.S.  Transportation  Command 
will  develop  the  capability  for  quantifying  the  percentage  of  logistics 
and  related  Automated  Data  Processing  (ADP)  systems  modernized 
using  implementation  status  as  of  the  end  of  FY  1999  as  a  baseline. 

For  each  major  system  undergoing  modernization,  track  annual 
progress  against  an  FY  2006  target  percentage.  Each  Component  will 
report  its  composite  progress  against  its  target  annually  beginning  at 
the  end  of  FY  2001. 

Minimize 
logistics  costs 
while  meeting 
Warfighter 
requirements 

Measure:  For  selected  fielded  weapon  systems,  reduce  the  logistics 
support  cost  per  weapon  system  per  year  compared  to  FY  1997 
baseline  as  follows:  7  percent  by  FY  2000;  10  percent  by  FY  2001 ; 
and  a  stretch  target  of  20  percent  by  the  end  of  F  Y  2005 . 

[After  Ref.  7] 

Table  8.  DoD  FY  2000  Logistic  Measures 


These  measures,  identified  in  Table  8,  correlate  to  Management  Information 
Systems,  two  of  which  are  strongly  related  to  system  readiness  and  availability.  These 
are:  1)  Optimize  support  to  the  Warfighter  and  2)  Implement  Customer  Wait  Time 
(CWT)  as  the  DoD  logistics  metric.  For  Management  Information  Systems,  availability 
and  readiness  are  measured  in  terms  of  system  down  time,  to  include  server,  workstation. 
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and  connectivity.  Measuring  mission  capability  for  Management  Information  Systems 
also  includes  system  down  time  and  the  processing  speed  for  the  system. 

To  reengineer  and  modernize  applicable  logistics  processes/systems  is  to  establish 
and  maintain  a  system  of  goals  and  measurements  for  the  Management  Information 
System.  Programs  can  achieve  this  through  development  and  periodic  assessment  of 
business  plans,  schedules,  objectives,  and  measures. 

O&M  funds  are  the  typical  type  of  funding  used  for  logistics  costs,  and  O&M 
dollars  are  a  scarce  resource  within  DoD.  Once  a  system  is  fielded,  the  manager’s 
primary  objective  is  to  minimize  logistics  costs  while  meeting  Warfighter  requirements. 
The  logistics  costs  for  fielded  Management  Information  Systems  includes  maintenance  of 
all  user  requirements,  upkeep  of  the  system,  and  iterative  maintenance  upgrades. 
Revalidating  the  validity  of  requirements  helps  to  control  costs.  Another  cost  control 
measure  for  logistics  is  consideration  of  the  system  architecture.  Application  of  new 
technologies  emerging  from  the  IT  industry  can  also  help  reduce  logistics  costs  to  fielded 
software  systems. 

2.  Industry  Measures  for  Fielded  Systems 

The  basis  for  most  measures  that  industry  uses  for  fielded  software  systems  is  the 
ability  to  earn  profit  on  sales  of  future  end  items  or  components,  to  improve  return  on 
investment.  Industry  wants  to  keep  the  customer  satisfied,  if  the  satisfaction  can  lead  to 
future  purchases  of  the  end  item  or  related  items.  In  commercial  industry,  companies 
also  want  to  measure  satisfaction  in  an  effort  to  improve  the  next  product  version  enough 
to  increase  customer  loyalty  and,  consequently,  improve  future  sales. 
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In  order  to  measure  and  improve  customer  satisfaction  for  fielded  software 
systems,  it  is  essential  to  understand  what  factors  actually  drive  customer  satisfaction. 
These  drivers  determine  attributes  to  measure  and  fimctions  or  processes  to  improve.  The 
concept  of  customer  satisfaction  in  industry  determines  future  customers  for  the  product 
or  service.  In  DoD,  most  users  do  not  have  the  choice  to  “shop  around.”  The  NICP  and 
Treaty  community  have  selected  CAMIN  as  the  system  that  the  sites  must  use. 
Nevertheless,  assessing  and  improving  customer  satisfaction  is  beneficial  to  maYimiVino 
efficient  and  timely  use  of  the  system  by  existing  users.  Customer  satisfaction  is  a  broad 
area,  encompassing  these  four  dimensions  of  program  management:  system  performance, 
customer  expectations,  customer  perception  of  his  or  her  interaction  with  the  PMO,  and 
problem  resolution. 

C.  CASE  INTERVIEWS 

Comparing  other  similar  fielded  software  systems  to  CAMIN  yields  insight  into 
the  program  management  measures  that  managers  typically  use.  The  following  two  DoD 
systems  have  characteristics  in  common  with  the  CAMIN  System.  These  systems  are  the 
Engineering  Data  Management  (EDM)  System  and  the  Federal  Emergency  Management 
Information  System  (FEMIS).  An  analysis  of  these  systems  provides  information 
concerning  the  systems,  including  a  description,  their  unique  parameters,  and  the  metrics 
used  in  program  management  of  these  systems. 
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1.  Engineering  Data  Management  System 

(Ref.  1) 

a.  System  Purpose 

The  Engineering  Data  Management  (EDM)  system  O&M  resides  within 
SBCCOM  to  maintain  and  provide  access  to  all  technical  data  for  the  command 
Research,  Development,  and  Acquisition  Center.  The  purpose  of  the  EDM  system  is  to 
manage  SBCCOM  engineering  data.  EDM  is  a  Product  Data  Management  (PDM) 
system,  and  PDM  is  the  commercial  name  for  the  generic  family  of  products  to  manage 
engineering  data.  Data  includes  the  engineering  drawings  and  specifications  and  the 
relationships  among  those  documents  to  include  configuration  management  relationships. 
The  system  controls  data  entry,  data  access,  and  associated  user  permissions. 

b.  System  Description 

The  EDM  is  both  thin  client  and  a  client-server  system.  The  user  interface 
is  completely  based  on  thin  client  architecture.  Thin  client  means  that  no  software  code 
or  data  are  stored  on  the  client  workstations.  The  client-server  system  predates  the  thin 
client,  to  reduce  operating  cost  and  complexity.  Once  thin  client  is  available  for  all  user 
needs,  the  client-server  system  serves  only  system  administration  and  data  administration 
fimctions. 

For  a  thin  client  system,  the  workstation  requirements  are  very  basic. 
These  COTS  software  packages  are  required:  a  128-bit  encrypted  web  browser,  Adobe 
Acrobat  Reader,  WordPad  or  Unix  text  reader,  and  a  raster  reader.  The  imager  for  the 
raster  reader  is  available  free  from  the  Army.  The  PDM  COTS  product  used  in  EDM 
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performs  these  functions.  It  manages  interfaces  and  data  access  for  the  Graphical  User 
Interface  (GUI)  system  administration  tool.  It  manages  the  vault  and  the  storage  media 
where  images  are  located.  Finally,  it  manages  the  Oracle  database  that  retains  the  Meta 
data  for  the  documents  stored  in  the  system. 

The  EDM  uses  three  different  types  of  thin  client  interfaces.  First,  the 
COTS  thin  client  interface  provides  all  the  same  capabilities  as  the  full  client  system, 
with  the  exception  of  system  administration  tools.  Permission  and  group  controls  permit 
operation  of  the  interface.  Second,  EDM  has  a  custom  interface  that  provides  a  simplistic 
view  and  access  to  the  system.  The  user-based  capabilities  of  the  web  interface  are  the 
same  as  the  full  system,  but  the  limited  features  simplify  navigation  within  the  system, 
and  users  find  this  interface  easier  to  leam  and  use.  Systemic  user  problems  drive  the 
need  for  this  interface.  When  users  are  not  accessing  the  system  fi-equently,  the  users  fail 
to  gain  the  familiarity  required  to  use  the  more  complex  version.  The  third  system  access 
portal  is  another  custom  interface.  This  interface  is  very  specialized,  providing  access  to 
selected  users  for  assignment  of  drawing  numbers.  Drawing  numbers  are  an  index  field 
in  the  system  database,  and  the  controls  must  exist  for  assignment  of  these  numbers. 

EDM  also  has  a  workflow  management  tool  as  part  of  the  PDM  COTS 
package.  The  workflow  management  tool  automates  the  engineering  data  processes.  It 
tracks  the  processing  and  status  of  the  necessary  events  to  generate  and  release  new  data. 

The  user  community  for  EDM  is  limited  to  DoD  employees  and 
contractors  that  require  access  to  the  data.  The  acquisition  community  uses  the  system  to 
provide  the  baseline  for  procurement  of  end  items  and  of  spare  and  repair  parts.  The 
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engineering  community  uses  it  as  a  repository  for  designs,  reference  source  for  design 
improvements  and  new  developments.  Engineers  also  use  the  system  to  provide 
information  in  support  of  investigations  of  field  failures.  In  the  current  configuration,  the 
system  is  simple  enough  that  no  user  manuals  or  formal  training  of  the  general  user  group 
is  required.  The  EDM  PM  operates  a  help  desk,  and  provides  straightforward  Avritten 
user  procedures,  upon  request. 

c.  System  Metrics 

The  PM  for  the  EDM  has  no  formal  program  for  metrics,  but  uses  the 
parameters  described  in  this  section  for  measuring  system  success.  EDM  measures  and 
assesses  customer  response.  The  PM  collects  these  data  regarding  the  satisfaction  of 
system  users,  mainly  through  interpretation  of  help  desk  calls.  The  help  desk  calls  may 
indicate  a  systemic  problem,  operational  problem,  or  training  problem. 

Another  measure  is  performance  against  operational  requirements. 
Services  provided  are  continuously  improved  based  on  the  PM  vision  of  what  a  PDM 
system  should  do  in  the  current  environment  and  what  tools  are  available  to  do  them.  A 
group  of  PMs  is  performing  similar  tasks  fi-om  other  AMC  Major  Subordinate 
Commands  (MSCs).  The  PM  for  this  system  uses  output  firom  the  group  to  codify  the 
vision.  The  result  is  a  performance  specification  for  the  Automated  Configuration 
Management  System  (ACMS)  [Ref.  1]. 

In  the  area  of  resource  management,  EDM  assesses  costs  and  bill-payers. 
The  EDM  PM  must  budget  well  in  advance  of  the  actual  need.  During  each  year,  the  PM 
uses  previously  budgeted  funding  to  contractor  consultant  hours  to  fix  bugs  and 
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implement  needed  enhancements.  Historical  data  helps  to  estimate  needed  funding. 
Annual  operating  cost  is  approximately  20  percent  of  the  initial  cost  to  field  an 
operational  system.  This  figure  correlates  to  the  annual  maintenance  fees  charged  by 
most  COTS  manufacturers. 

The  EDM  has  two  types  of  paying  customers.  Project  teams  that  are 
requiring  engineering  data  to  be  stored  in  the  repository  pay  variable  costs.  The 
SBCCOM  RDA  Enterprise  pays  the  fixed  costs,  so  that  the  engineering  and  acquisition 
community  can  use  the  system  to  access  the  data.  These  customers  are  widely  satisfied 
with  the  system,  and  budget  to  maintain  the  system  at  required  levels. 

Quality  is  another  concern  of  the  EDM  program  manager.  The  PMO  tests 
each  software  change,  and  performs  troubleshooting  when  a  problem  is  suspected.  They 
maintain  a  test  bed  to  facilitate  regular  testing.  Testing  methods  consist  mostly  of  testing 
various  Use  Cases  and  reporting  problems  back  to  software  consultants  for  correction. 

2.  Federal  Emergency  Management  Information  System 

(Ref  12  and  13) 

a.  System  Purpose 

The  SBCCOM  and  the  Federal  Emergency  Management  Agency  jointly 
developed  the  FEMIS.  The  FEMIS  is  an  integrated  system  that  provides  planning 
coordination,  response,  and  exercise  support  for  emergency  management  personnel. 

During  an  emergency  response,  the  FEMIS  users  retrieve  and  execute 
plans  created  under  non-emergency  conditions.  FEMIS  uses  real-time  data  from  outside 
sources,  such  as  meteorological  monitors.  To  support  the  decision-making  process. 
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FEMIS  provides  these  outputs  to  the  user:  resource  tracking,  task  lists,  contact  lists,  event 
logs,  status  boards,  hazard  modeling,  and  evacuation  modeling. 

b.  System  Description 

Local,  state,  and  Federal  emergency  management  experts  developed  the 
requirements  for  FEMIS.  Users  were  actively  involved  in  the  prototype  reviews,  testing, 
and  setting  priorities.  Continuing  development  is  the  result  of  user  input  and  review. 
FEMIS  meets  Federal  and  industry  open  systems  standards.  The  system  architecture 
consists  of  a  suite  of  COTS  software  products,  including  ArcView,  Oracle,  and  Microsoft 
Project.  The  U.S.  Government  provides  the  hazard  analysis  and  evacuation  models. 

The  system  platform  is  client-server  based  to  support  multiple  users, 
distributed  data,  and  multiple  operations  centers.  FEMIS  uses  a  UNIX  central  data  server 
and  personal  computer  clients  with  Windows  NT.  The  size  of  the  FEMIS  is 
approximately  1.25  million  SLOCs.  The  system  has  up  to  350  users.  Leased  lines 
coimect  system  nodes.  The  system  uses  a  LAN  connection  from  client  to  server. 

The  FEMIS  system  development  is  recent.  Requirements  definition 
occurred  in  1993.  The  first  system  fielding  occurred  in  1995  as  version  1.1.  General 
system  fielding  occurred  in  1999  as  version  1.4.5.  The  current  version  is  1.4.7. 
Maintenance  releases,  with  enhancements,  occur  about  every  8-12  months. 

The  PM  for  FEMIS  is  evaluating  the  conversion  of  the  user  interface  to  a 
thin  client  structure.  A  conversion  would  enhance  the  FEMIS,  reducing  firewall  issues, 
improving  system  accessibility,  and  saving  life  cycle  costs. 
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c.  System  Metrics 

The  FEMIS  PM  uses  the  Software  Enhancement  and  Problem  Report 
(SEPR)  database,  maintained  by  the  contractor,  to  measure  the  system.  The  reports  track 
SEPRs  by  three  aspects:  severity,  time  to  close,  and  rate  in  versus  rate  closed.  The  SEPR 
process  introduces  all  enhancements  to  the  system.  The  PM  uses  the  following  two 
measures  for  tracking  SEPRs:  severity  measure,  which  is  commercial  practice,  and 
measure  of  DoD  priority. 

D.  PDSS  CANDIDATE  METRICS 

This  section  provides  descriptions  of  the  candidate  metrics  for  fielded  software 
systems.  Table  9  summarizes  the  metrics  from  this  chapter,  and  cross-levels  among  the 
possible  sources  of  metrics,  to  include:  1)  software  metrics,  2)  fielded  system  metrics,  3) 
metrics  derived  from  the  case  analysis,  and  4)  CAMIN  metrics.  The  metrics  collected 
vary  in  scale,  and  major  metrics  encompass  minor  metrics.  The  combination  of  metrics 
from  these  various  sources  enriches  the  definition  of  the  metrics  with  respect  to  reality  in 
a  PDSS  environment.  This  section  also  discusses  each  metric  and  its  usefulness.  Table 
10,  presented  near  the  end  of  this  chapter,  encapsulates  the  detailed  list  of  PDSS 
candidate  metrics  and  the  associated  measures,  derived  from  of  this  research. 

1.  Management  and  Planning 

Measuring  and  tracking  costs  and  schedule  are  important  to  ensure  that  a  program 
remains  within  existing  funding  and  meets  schedule  requirements.  These  measures  help 
the  PM  to  predict  cost  and  schedule  of  the  fielded  software  systems  for  future  years,  and 
to  help  identify  areas  for  cost  control.  The  funding  rates  identify  problem  areas  and 
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opportunities  for  diversion  of  funding  from  lagging  areas  to  more  urgent  and  important 
ones.  Counting  hours  expended  is  a  measure  of  trends  and  assessment  of  efficiency. 
Software  design  and  development  is  an  art  as  well  as  a  science,  and  writing  new  code 
may  not  adhere  to  a  strict  schedule.  As  a  result,  slips  in  schedule  may  affect  available 
funding. 

The  PM  uses  measures  to  assess  the  cost  expenditures  for  software  tasks, 
compared  to  the  initial  cost  estimates.  Program  cost  measures  assess  trends  and 
improvements,  and  provide  data  for  future  cost  estimates.  PMs  need  to  track  obligated 
and  expended  costs  against  progress  and  number  of  defects.  Charts  and  graphs  of  these 
data  improve  the  data  analysis  process.  Schedule  charts  that  managers  can  use  to 
measure  progress  include  Gantt  and  PERT  charts. 

The  Constructive  Cost  Model  (CoCoMo  II)  [Ref  6]  is  one  method  for  overall 
management  when  considering  cost,  schedule,  and  performance.  CoCoMo  II  provides  a 
range  on  its  cost,  effort,  and  schedule  estimates,  from  best  case  to  most  likely  to  worst- 
case  outcomes. 

It  is  important  to  remember  that  software  programs  are  notorious  for  going 
beyond  schedule  and  over  budget.  Planning  provides  a  systemic  approach  to  controlling 
schedule  and  other  program  problems.  A  strategic  plan  for  the  system  with  objectives, 
goals,  and  measures  becomes  a  strong  baseline  for  achieving  control. 


63 


Soft^vare  Metrics  MetriS^^™  f*'®*"  CAMIN  Metrics 


Cost  and  Schedule, 

j  Software  Cost,  Planning 

Engineering  Levels 

Environment 


Computer  Resource 
2  Utilization, 
Manpower,  Effort 


Requirements 
-  Traceability  and 
Stability,  Fault 
Profiles 


Design  Stability, 
Complexity,  Size, 
Progress,  Breadth 
and  Depth  of 
Testing 


Cost,  Quality 


Funding  Obligated 
and  Disbursed, 
Schedule,  Program 
Management, 
Quality  Assurance 


Critical  Resource 


Requirements 

Validation 


Rate  in  versus  rate  „  . 

closed,  Severity,  PCR, 

Time  to  Close,  DoD  Configuration 
priority  Management 


Fault  testing 


5  Reliability 


System  Readiness 
and  Availability 


Problem  Resolution 


System  Performance  System  Performance 
against  requirements 


Development, 
Percent  Completion, 
Peer  Review 


User  Accounts  and 
Logins,  Help, 
Training,  User, 
Hardware,  Software, 
Training, 
Maintenance 


Data  Interventions, 
Actions,  Intergroup 
Coordination 


Customer 
Satisfaction, 
Expectations, 
Perception  of  PM 


Customer  Response 
and  Satisfaction 


[Developed  by  Researcher] 

Table  9.  Candidate  PDSS  Metrics 
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The  Software  Engineering  Environment  (SEE)  metric  is  a  measure  that  is  oriented 
toward  software  development,  but  has  application  for  fielded  software  systems.  This 
measure  rates  the  developer's  application  of  software  engineering  principles.  Examples 
of  these  principles  are  the  use  of  structured  design  techniques,  the  extent  of  tool  usage, 
and  the  use  of  requirement  management  techniques.  A  qualified  independent  group 
should  perform  rating  of  software  developers.  One  example  of  this  is  the  SW-CMM 
[Ref  20]  developed  by  the  SEI  at  Camegie-Mellon  University. 

The  results  of  auditing  software  quality  assurance  provide  an  estimation  of 
product  quality  and  compliance  of  staff  to  the  internal  processes.  A  review  of  results 
provides  the  status  of  action  items  from  life-cycle  reviews.  Trouble  reports  provide 
insight  into  the  quality  of  the  product  and  processes,  and  the  effectiveness  of  testing. 
Peer  review  results  provide  insight  into  the  quality  of  the  intermediate  and  final  products 
and  into  the  peer  review  and  development  processes.  Defect  prevention  provides  insight 
into  the  cause  of  defects  and  the  effect  of  defect  prevention  activities  on  defect  insertion 
rates.  Quality  assurance  is  a  measure  of  how  compliant  the  program  staff  is  performing 
activities  associated  with  the  SEE. 

2.  Resource  Utilization 

i 

Resources  utilized  for  a  program  include  computer  resources,  equipment,  and 
staff.  It  is  beneficial  to  maintain  utilization  of  these  resources  at  a  constant  level. 
Measurement  of  these  resources  over  time  provides  indicators  of  fluctuation  of  activity 
and  inefficient  operation. 
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Computer  resource  utilization  (CRU)  measures  performance  with  respect  to  its 
computer  resource  utilization  goals  and  requirements.  Measures  of  computer  resource 
utilization  provide  another  tool  for  measuring  trends  in  costs.  This  measure  monitors 
utilization  of  development  resources. 

The  staff  and  their  equipment  are  the  critical  resources  of  the  fielded  software 
systems.  The  PM  can  optimize  costs,  expenditures,  and  schedule  through  awareness  and 
management  of  these  resources.  Measure  of  effort  provides  visibility  into  the 
contribution  that  staffing  has  on  project  costs,  schedule  adherence,  and  product  quality. 
This  measure  helps  the  PM  to  assess  current  cost  expenditures  and  anticipate  future  costs. 
This  metric  provides  an  indication  of  the  developer's  application  of  human  resources  to 
the  development  program  and  the  ability  to  maintain  sufficient  staffing  to  complete  the 
project.  The  manpower  metric  is  composed  of  two  parts:  an  effort  measure  monitors 
labor  hours  planned  and  expended,  while  a  staffing  measure  accounts  for  quantity  and 
types  of  personnel  needed  to  do  the  work.  This  metric  assists  the  Government  in 
determining  whether  the  developer  has  scheduled  a  sufficient  number  of  employees  to 
produce  the  product  in  the  time  and  budget  allotted. 

3.  Requirements  and  Configuration  Management 

This  metric  provides  for  the  assessment  of  requirements  traceability,  validation, 
evaluation,  tracking,  and  implementation.  Changes  to  requirements  are  inevitable,  and 
drive  up  the  life  cycle  cost  of  fielded  software  systems.  Maintaining  comprehensive 
measures  of  requirements  changes  helps  to  control  costs  and  maintain  stability. 
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The  requirements  traceability  metric  measures  the  portion  of  the  software  system 
that  has  implemented  the  documented  system  reqmrements.  Also  measured,  is  how  well 
the  system  requirements  align  with  requirements  in  higher-level  docvunents.  The 
software  system  includes  software  code  and  associated  items,  including  specifications, 
software  design,  code,  and  test  cases.  Tracking  System  Requirements  provides  an 
historical  record. 

Configuration  management  for  fielded  software  systems  includes  submission  and 
processing  of  change  requests.  Change  requests  are  an  important  source  of  user 
requirements.  Measures  associated  with  configuration  management  help  to  assess  trends 
in  requirements  and  progress  to  accomplish  these  requirements.  Users,  funding 
managers,  or  data  customers  typically  establish  priorities  and  severity/importance  for 
change  requests  during  the  CCB.  When  users  present  change  requests,  the  response  time 
may  be  important.  Changes  to  CAMIN  through  Version  updates  can  take  up  to  two  years 
fi-om  contract  task  to  fielding. 

4.  Development  and  Maintenance  Upgrades 

The  degree  of  difficulty  of  development  and  maintenance  upgrades  is  dependent 
on  the  size,  complexity,  and  stability  of  the  software  architecture  and  code.  This  metric 
addresses  the  entire  process  to  develop  and  modify  software,  including  design,  code,  test, 
and  field. 

Software  that  is  more  complex  is  harder  to  understand,  test  adequately,  and 
maintain.  Additionally,  a  highly  complex  unit  has  a  greater  tendency  to  contain 
embedded  errors  than  a  unit  of  lower  complexity.  The  likelihood  of  introducing  errors 
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when  making  code  changes  is  higher  in  complex  units.  The  design  stability  measure 
evaluates  changes  made  to  the  design  of  the  software.  The  complexity  metric  provides  a 
means  to  measure  and  evaluate  the  structure  of  software  units. 

Size  measures  are  used  to  estimate  cost,  schedule,  and  workload.  Size  of  software 
implies  the  scale  of  the  system.  Measures  of  size  include  source  lines  of  code  (SLOCs), 
Function  Points,  and  Feature  Points.  A  measure  of  the  size  of  a  system  can  vary 
depending  on  method  of  measure.  The  purpose  for  measuring  size  is  to  define  the 
complexity  of  a  system  and  how  long  it  takes  to  write  or  change  the  system.  One 
measure  is  to  count  the  number  of  lines  of  code,  called  “source  lines  of  code,”  or  SLOG. 
The  software  community  accepts  this  method,  especially  for  use  with  older  programming 
languages.  Newer  programming  languages,  including  those  based  on  Object-Oriented 
logic,  do  not  compare  through  lines  of  code.  The  only  way  to  measure  the  size  of  a 
system  for  comparison  purposes  is  to  establish  function  point  or  feature  point  measures. 
Further,  system  size  provides  useful  comparison  only  when  measured  consistently.  It  is 
important  to  be  consistent  in  the  method  of  counting  to  ensure  common  interpretation  of 
the  measure.  The  International  Function  Point  User  Group  (IFPUG)  [Ref  15]  has 
established  basic  counting  methods  to  provide  consistency  in  the  methodology. 
However,  older  systems  do  not  typically  use  function  point  analysis,  so  comparison  is 
impossible.  The  DoD  continues  to  require  SLOG  as  the  standard  measure  of  system  size, 
whereas  function  and  feature  points  include  complexity  ratings  and  other  factors. 
Measuring  size  of  software  is  very  controversial  due  to  technological  advances  in  code 
implementation.  The  basis  for  controversy  is  comparing  items  of  like  size.  Systems 
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based  on  object-oriented  design  principles  are  not  comparable  to  systems  designed  using 
functional  principles.  In  object-oriented  systems,  duplication  within  classes  means  that  a 
system  may  reuse  lines  of  code.  A  primary  feature  of  Object-Oriented  design  is  to  reduce 
complexity,  maximize  reuse,  and  minimize  problems  associated  with  future  maintenance 
and  changes. 

Measuring  progress  is  to  assess  the  degree  of  certainty  to  complete  the  project  on 
schedule.  Progress  is  a  measure  of  how  well  the  project  is  performing  with  respect  to  its 
schedule.  The  development  progress  metric  measures  the  degree  of  completeness  of  the 
software  development  effort,  indicating  the  readiness  to  proceed  to  subsequent  activities 
in  software  development.  One  should  begin  collecting  data  during  software  requirements 
analysis  and  continue  throughout  software  development  and  fielded  software  systems. 

The  schedule  and  progress  regarding  work  completion  is  a  key  program 
management  tool  for  a  software  development  or  modification  project.  Developers  and 
the  PMO  should  negotiate  the  schedule  to  arrive  at  a  realistic  plan.  For  initial 
development  and  new  developers,  the  estimate  for  time  is  usually  underestimated.  For 
developers  that  have  specific  program  experience,  as  those  on  CAMIN,  estimates  for  time 
and  cost  are  very  close  to  execution. 

PMs  assume  program  risk  if  the  software  development  tasks  are  primarily  driven 
by  schedule.  Results  often  include  reduced  quality,  reduced  documentation,  and  missed 
deadlines.  When  schedule  becomes  an  issue  for  release  of  a  software  modification 
package  for  the  CAMIN  program,  the  list  of  tasks  included  in  a  software  modification  is 
reduced,  rather  than  pushing  the  developers  to  meet  an  unrealistic  schedule. 
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Development  includes  all  of  the  iterative  tasks  involved  in  the  design  and  writing 
of  code.  For  the  CAMIN  program,  the  PM  does  not  monitor  to  this  level  of  detail. 
Instead,  the  PM  monitors  the  processes  used  and  certifications.  The  users  assess  the 
output  of  code  during  final  acceptance  testing  before  release  of  version  update  software. 

The  CAMIN  contractor  performs  use  cases  and  regression  analyses  to  measure 
defect  rates  and  correct  problems.  Use  cases  are  scenarios  that  depict  how  the  system  is 
used.  Ideally,  there  would  be  at  least  one  use  case  for  every  possible  use  of  the  system, 
and  testing  would  find  every  defect.  For  complex  systems  like  CAMIN,  use  case  analysis 
is  very  time  consuming  and  therefore  expensive. 

The  PM  should  run  independent  testing  of  the  system.  The  tests  should  occur  at 
each  system  modification,  before  the  PM  approves  the  software  for  release.  For  CAMIN, 
as  a  Management  Information  System  with  a  relatively  small  user  group,  a  user-based  test 
occurs.  The  PMO  prepares  a  test  plan  that  includes  evaluation  of  requirements  from  the 
^^tial  tasking.  The  test  plan  includes  scenarios  that  verify  the  change  is  in  accordance 
with  the  task.  Developers  systematically  fix  and  retest  defects  found  during  user  testing. 
Occasionally,  the  PM  and  contractor  may  negotiate  a  low  priority  defect  that  would  delay 
fielding  into  a  later  release.  During  testing  for  CAMIN  modifications,  the  number  of 
defects  found  is  considered  low,  as  it  ranges  from  none  at  the  least  to  two.  Typically, 
user  testing  also  yields  the  submission  of  30-40  new  change  requests. 

The  breadth  of  testing,  also  known  as  "black  box"  testing,  evaluates  the  testing 
performed  on  the  system  from  the  user  perspective.  This  metric  is  concerned  with 
obtaining  correct  outputs  because  of  prescribed  inputs.  No  test  can  cover  every  possible 
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combination  of  variables  involved  in  a  Management  Information  System.  Variables 
include  interface,  platform,  and  use  case. 

The  depth  of  testing,  also  known  as  "white  box"  testing,  measures  the  amoimt  of 
testing  achieved  on  the  software  architecture.  This  metric  focuses  on  the  visibility  into 
how  the  software  is  constructed. 

5.  Availability 

This  metric  encompasses  the  factors  that  keep  the  system  ready,  operational, 
accessible,  and  user-fiiendly.  Important  components  of  this  metric  are  user,  system,  and 
data  administration,  along  with  user  support  factors  such  as  training  and  help  line.  The 
reliability  metric  tracks  system  failures  caused  by  software  and  the  time  it  takes  to  restore 
the  system  to  its  previous  operating  condition  after  these  failures  occur.  User,  system, 
and  data  administration  are  areas  of  CAMIN  that  the  PM  uses  to  control  costs,  ensure  the 
system  functions,  satisfy  customers,  and  protect  system  data. 

The  user  list  for  CAMIN  frequently  changes.  Usage  is  the  basis  for  hardware  and 
software  purchases,  and  seat  licenses  to  use  software.  Security  advises  administrators  to 
deactivate  a  user  accoimt  if  the  user  is  not  using  the  system.  Control  of  user  accounts 
also  saves  administrative  costs.  Measures  have  identified  CAMIN  workstation  users  who 
only  use  capabilities  that  are  available  on  the  CAMIN  web  site.  The  PM  revoked 
workstation  accoimts  imtil  needed.  These  measures  also  provide  usefiil  data  to  justify 
adding  new  thin  client  capability. 
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Training  provides  users  with  the  tools  and  techniques  for  using  the  CAMIN 
system.  For  the  complex  operations  in  CAMIN,  users  need  to  repeat  training  after  long 
periods  of  system  disuse. 

User  help  provides  an  instant  tool  for  CAMIN  users.  As  discussed  with  the  case 
interviews  in  this  chapter,  measures  of  user  help  provide  insights  into  problems  with  the 
system,  with  the  user  manual,  and  the  training  program. 

User  Hardware/Software  List  keeps  track  of  the  amoimt  and  types  of  software  and 
hardware  in  the  inventory.  The  amount  of  hardware  and  software  fielded  drives  up 
maintenance  costs.  The  amount  and  configuration  of  these  products  affect  system 
functionality. 

The  system  maintenance  activities  improve  efficiency  of  system  operations.  For  a 
complex  system,  these  activities  may  be  time-consuming.  However,  the  actions 
addressed  in  this  section  are  required  for  maintaining  system  operations  and  addressing 
customer  issues. 

6.  Problem  Resolution 

The  area  of  customer  satisfaction  is  the  in-depth  exploration  of  customer 
problems.  The  goal  is  to  create  an  environment  where  respondents  feel  free  to  report  and 
describe,  in  detail,  problems  they  have  experienced.  After  learning  about  specific 
customer  problems,  it  is  essential  to  follow  up  and  determine  how  well  these  problems 
were  resolved. 

Problem  resolution  is  crucial  to  maintaining  customer  satisfaction.  Most 
customers  recognize  that  occasional  problems  are  unavoidable  and  even  inevitable. 
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However,  the  method  and  attitude  that  the  staff  uses  to  respond  to  those  problems  is  often 
the  difference  between  a  satisfied  and  unsatisfied  customer. 

Good  problem  resolution  can  increase  customer  satisfaction  beyond  the  level  that 
existed  before  the  problem  occurred.  Customers  who  report  that  a  company  has  exceeded 
their  expectations  fi'equently  cite  quick,  customer-oriented  problem  resolution  as  the 
source  of  their  satisfaction. 

7.  Performance  against  Requirements 

Measuring  the  performance  of  a  system  is  necessary  to  determine  customer 
satisfaction.  Defined  user  requirements  provide  the  benchmark  for  measuring  system 
performance,  and  the  requirement  baseline  is  always  in  flux.  System  testing  through 
IV&V  and  operational  testing  provide  measures  of  performance.  These  tests  must  be 
repeated  periodically  to  correspond  to  changes  to  the  requirements,  environment^  or 
system. 


8.  Customer  Satisfaction 

Understanding  customer  expectations  and  then  meeting  or  exceeding  those 
expectations  is  fundamental  to  creating  satisfaction.  Customers  become  satisfied  only 
when  the  system  meets  or  exceeds  expectations.  It  is  important  to  recognize  that 
expectations  are  not  static.  Performance  that  satisfies  a  customer  today  may  not  be 
sufficient  to  satisfy  that  same  customer  in  tomorrow's  environment.  In  addition,  the  user 
group  is  in  a  state  of  flux  and  users  with  different  experiences  may  have  different 
expectations. 
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A  customer's  perception  of  his  or  her  interactions  with  the  PMO  is  another  key 
driver  of  customer  satisfaction.  Quite  frequently,  how  customers  feel  about  their 
treatment  is  more  important  than  the  underlying  quality  of  the  product  or  service.  Poor 
treatment  leaves  a  damaging  and  lasting  impression  that  is  difficult  to  overcome. 

The  user  expects  a  system  that  meets  his  or  her  performance  requirements  and  is 
easy  to  use.  In  the  case  of  the  CAMIN  system,  the  IV&V  was  conducted  with  the  intent 
of  measuring  the  system  against  the  full  program  requirements.  To  maintain 
configuration  management,  the  requirements  database  maintains  old  and  new  system 
requirements.  New  requirements  come  from  approved  PCRs  and  from  program 
management  decisions.  At  each  version  release,  the  PM  and  contractor  jointly  conduct 
extensive  testing  on  parts  of  the  system  that  may  have  been  impacted.  The  user, 
however,  develops  and  changes  expectations  of  a  system.  Most  users  are  accessing  other 
systems  that  they  naturally  compare.  The  other  systems  may  respond  more  quickly, 
making  this  system  seem  slow.  The  other  system  may  provide  a  simpler  interface, 
causing  users  to  request  the  same  short  cuts  and  interfaces.  The  users  may  expect  to 
work  in  a  familiar  environment,  such  as  Microsoft  Windows  or  web  browser.  Thus, 
using  differing  computer  systems,  users  develop  expectations  about  speed  and  ease  of 
access  or  operation. 
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Metric 

Measures 

1 

Management 
and  Planning 

Cost  and 
Schedule, 

Funding 

Obligated  and 
Disbursed, 
Schedule,  SEE, 
Planning  Levels, 
Quality  Program 
Management, 
Quality 

Assurance 

Computed  maturity  level 

Measures  quality  of  procedures  against  published  procedures 
Quality  Audits  planned/conducted 

Nonconforming  conditions  found/closed/Open 

Monitor  Strategic  Plan 

Measure  adequacy  of  funding 

Measure  of  costs  by  year,  and  by  amount  of  requirement  or 
function  point 

Measure  of  funding  obligations  and  dispersals  vs.  plan  (Dollars 
spent  vs.  dollars  allocated) 

Measure  of  hours  expended  vs.  hours  forecast  and  requirements 
Hours  expended  by  day,  month,  task  order/project 

Cost  expended  by  month,  task  order/project 

Baseline  (cost/hours/duration)  vs.  actual  &  progress  (% 
complete) 

2 

Resource 

Utilization 

CRU, 

Manpower, 

Effort, 

Percent  resource  capacity  utilized 

Measure  of  staff  levels  and  rate  of  changes  by  function 

Assess  funding  and  personnel  resources  vs.  work  to  be 
performed 

Measure  of  Computer  Resource  Utilization  (CRU)  vs. 
requirement 

Staffing  /  Equipment  resources  needed  over  lifecycle 

3 

Requirements 

and 

Configuration 

Management 

Requirements 
Traceability  and 
Stability, 
Validation,  Fault 
Profiles,  Rate  in 
versus  rate 
closed.  Severity, 
Time  to  Close, 
DoD  priority 
Requirements, 

PCR 

Measure  of  problems  /  defects  over  time  and  over  versions 

Number  of  requirements  changes 

Measure  of  priority/Measure  of  severity 

Measure  of  difficulty 

Importance  of  change  vs.  Rate  Closed 

PCR  Rate  in  vs.  Rate  Closed 

Requirements  Identified  (Percent  requirements  traced),  fulfilled, 
outstanding,  validated  (for  a  release) 

Requirements  by  release,  by  configuration  item,  by  category 
(new,  baseline  change,  design  change,  implementation  change) 
Changes  per  configuration  item  (volatility) 

Backup  success.  Backup  resources  used 

Hardware/  Software  Environment  Changes 

Number  of  files  checked  out/Number  of  files  changed 

Number  of  users  who  checked  out/changed  a  file 

Number  of  times,  length  of  time  each  file  was  checked  out 

Number  of  lines  that  changed  per  file  (code  only) 

Average  number  of  changed  lines  (code  only) 
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4 

Development 

and 

Maintenance 

Upgrades 

Design  Stability, 
Complexity, 

Size,  Progress, 
Breadth  and 
Depth  of  Testing 
Fault  testing. 
Percent 

Completion,  Peer 
Review 

Cost  to  complete.  Hours  to  complete.  Milestone/event  slippage 
Software  size  by  version  (SLOC),  by  tasking  (function  point) 
Periodically  assess  the  development,  documentation,  peer 
review,  and  testing  procedures  (use  SEE,  where  possible) 

Breadth,  Depth  of  Testing,  Degree  of  code  testing 

Count  of  defects  foimd  during  testing 

Stability  and  Complexity  indices 

Percent  requirements  tested  /  passed 

Number  and  types  of  faults,  average  open  age,  restoration  times 
Time  to  implement  (baseline  vs.  actual)  by  software 
change/Configuration  Item 

Defects/Problems  identified,  corrected,  validated,  time  and  effort 
to  correct 

Peer  Reviews  planned/conducted 

5 

Availability  and 
Usage 

Reliability, 

System 

Readiness,  User 
Accounts  and 
Logins,  Help, 
Training,  User, 
Hardware, 
Software, 

Training, 

Maintenance 

Time  system  is  not  available  for  planned  and  unplanned  outages 
Monitor  Continuity  of  Operations  Plan 

User  vs.  system  usage.  New  users,  and  deactivated  users 

Trainee  feedback  on  availability,  quality  of  training 

Last  user  training  vs.  error  frequency  for  user 

Categorize  help  calls  to  identify  trends 

Measure  frequency  of  user  help  vs.  date  of  last  training 

Measure  of  system  and  document  usage 

Count  and  categories  of  maintenance  issues 

Count  and  time  to  resolve  non-conforming  conditions 

Mean  time  between  failures 

Effort/hours  expended  in  staff  familiarization  (training) 

6 

Problem 
Resolution  Data 
Interventions, 
Actions, 

Intergroup 

Coordination 

Measure  actions  and  resolutions,  time  spent,  type  of  problems 
Measure  rate  of  actions  vs.  time  to  close,  based  on  priority 

Problem  Calls  logged  (by  category,  by  users,  by  action) 

Time/effort  to  service  calls 

Open  problem  calls 

Data  changes/interventions  logged 

Time/effort  to  make  data  change 

7 

Performance 

against 

Requirements 

Compare  system  performance  to  requirements  documents 

Measure  system  performance  against  defined  requirements 

8 

Customer 

Satisfaction 

Fertorm  surveys  for  user  satisfaction,  user  meetings,  help  line, 
satisfaction  with  program  management  staff,  overall  satisfaction 

[After  Table  7,  Table  9,  Appendix  D] 

Table  10.  PDSS  Metrics  and  Measures 


76 


The  concepts  of  customer  satisfaction  in  industry  determine  future  customers  for 
the  product  or  service.  In  DoD,  most  users  do  not  have  a  choice  to  “shop  around.”  The 
NICP  and  Treaty  community  have  selected  CAMIN  as  the  system  that  participating  sites 
must  use.  Nevertheless,  assessing  and  improving  customer  satisfaction  is  beneficial  to 
encovurage  users  to  utilize  the  system  and  user  support  systems,  such  as  the  help  line  and 
manuals,  and  maintain  timely  and  correct  system  data.  The  users  interact  with  the  PMO 
through  the  help  line,  meetings,  telephone  calls,  and  e-mails.  The  result  is  that  the  degree 
of  user  satisfaction  determines  how  often  the  user  utilizes  the  staff  resources. 

Most  users  present  problems  through  the  help  line.  The  staff  must  then  follow-up 
help  requests  to  be  sure  that  the  user  considers  the  problem  closed,  and  is  satisfied  with 
the  outcome.  Data  collected  in  the  help  line  database  can  provide  access  to  useful 
measures.  It  is  also  important  to  provide  a  mechanism  for  users  to  provide  anonymous 
feedback  on  their  perception  of  help  provided. 

9.  Final  Metrics  List 

The  metrics  and  measures  collected  in  this  chapter  provide  a  good  baseline  for 
analysis  of  systems  in  the  PDSS  phase.  Table  10  provides  a  summary  of  the  metrics  and 
measures,  consolidated  by  the  eight  categories  that  agree  ■with  Table  9.  This  list  may 
prove  useful  in  analysis  of  various  PDSS  that  may  be  vastly  different  in  design, 
application,  or  management. 
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IV.  DATA  ANALYSIS  OF  PDSS  METRICS  FOR  CAMIN 


This  chapter  analyzes  data  collected  in  research  for  this  thesis.  The  background 
presented  in  Chapter  II  contains  a  concise  list  of  the  CAMIN  program  management 
issues.  Data  from  Chapter  III  generates  a  list  of  candidate  metrics  from  DoD  and 
commercial  sources,  and  from  other  fielded  software  systems.  Through  analysis  of  these 
data,  this  chapter  develops  a  list  of  the  metrics  that  are  recommended  for  use  by  the 
CAMIN  program  management. 

The  analysis  consists  of  evaluating  key  component  areas  for  each  issue  from 
Chapter  II,  and  identifying  the  parameters  that  require  metrics  and  measures.  Finally,  this 
chapter  analyzes  lessons  learned  from  a  review  of  the  CAMIN  program,  data  collection, 
and  the  case  studies  contained  in  this  thesis. 

A.  SELECTION  OF  METRICS  AND  MEASURES  FOR  THE  CAMIN 

This  section  analyzes  the  issues  to  select  metrics  for  use  by  the  CAMIN  PMO. 
Table  1 1  smnmarizes  these  metrics. 

1.  Funding 

The  issue  of  funding  ties  into  all  of  the  other  program  management  issues 
addressed  in  this  thesis.  Chapter  II  discusses  problems  that  arise  if  the  program  funding 
is  less  than  the  requirement  in  a  given  year,  and  over  the  long  term.  Table  1 1  shows  the 
metrics  and  measures  that  serve  to  address  the  programmatic  flmding  issues  for  CAMIN. 
Measuring  success  in  funding  includes  other  factors  that  help  the  PM  to  successfully 
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request  and  receive  that  funding.  Following  is  a  detailed  discussion  of  each  of  these 
metrics,  the  measures,  and  methods  of  calculation. 

a.  Understanding  Funding  Needs 

The  PMO  must  evaluate  each  funding  driver  of  the  CAMIN  program  to 
determine  if  it  is  a  valid  need.  The  reason  for  this  is  that  organizations  responsible  for 
funding  the  CAMIN  system  require  revalidation  of  the  need  for  requirements.  All 
aspects  of  the  program  drive  funding,  so  measuring  all  areas  provides  data  to  understand 
funding  needs. 

The  methods  for  evaluating  funding  drivers  include  the  time  and  cost  of 
implementing  the  new  requirement,  which  may  determine  the  validity  of  a  requirement. 
If  implementation  is  prohibitively  expensive,  the  alternative  to  use  a  manual  or  other 
method  may  be  preferred.  However,  there  are  intrinsic  benefits  to  implementing  new 
requirements  that  may  not  lend  themselves  to  cost-benefit  analysis.  The  PM  must 
consider  user  need,  mission  accomplishment,  data  access  and  protection,  productivity  and 
retention  of  staff  and  resources,  and  the  specific  needs  defined  by  the  organizations  that 
control  the  fimding  when  determining  the  validity  of  a  new  requirement. 

Another  method  for  evaluating  fimding  drivers  involves  measuring  the 
justification  for  funding  needed  to  resolve  PCRs,  which  comes  firom  the  CCB  evaluation 
of  change  requests.  The  CCB  evaluation  measures  breadth  of  customer  need'benefit,  cost 
and  timefi-ame  of  meeting  the  requirement,  and  the  direct  benefit  to  system  fimctionality , 
as  well  as  the  ability  to  meet  the  mission.  The  PM  must  retain  CCB  evaluation  data  to 
support  future  actions  and  to  assess  trends. 
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METRICS 


MEASURES 


Understanding  Funding  Needs 

Measure  each  funding  driver  for  the 
priority/severity  of  need  and  for  the 
likelihood  of  funding  acceptance 

Funding 

Communication  of  Funding 
Justification 

Measure  the  degree  of  understanding  by 
the  target  audience 

Receipt  of  Required  Funding 

Measure  and  track  the  adequacy  of 
funding  in  each  area 

Power  to  the  System 

Measure  the  reduction  in  system 
availability  due  to  power  failures 

System 

Connectivity 

Measure  the  consistency  and  effectiveness 
of  cormectivity 

Availability 

Workstation/User  Accessibility 

Measure  the  ability  of  users  to  access  the 
CAMIN  Server 

System  Maintenance 

Measure  the  frequency,  nature,  and 
impacts  of  maintenance  activities 

User  Competency  and 
Preparedness 

Measure  PM  actions  that  assist  user 
competency 

Data  and 

Outnut 

CAMIN  Software  Design 

Measure  the  design  characteristics  that 
cause  CAMIN  to  meet  requirements 

Accuracy 

Data  Audit 

Measure  the  frequency  and  quality  of 
audits  to  determine  system  data  accuracy 

Data  Interventions 

Measure  the  frequency,  nature,  method 
and  time  spent  on  data  interventions 

Experienced  IT 
Support 

Leveling  of  Tasks 

Measure  the  amount  and  types  of  work 
under  the  contract  and  the  contract  vehicle 

Challenges  and  State  of  the  Art 

Measures  the  trends  in  technology 

Requirements 

Changes 

Changes  to  External 

Environment 

Measure  the  COTS  software  and  hardware 
and  the  network  environment  to 
proactively  adjust  to  changes 

Evaluation  of  Changes 

Measure  the  priority,  type,  complexity, 
and  severity  of  change  requests 

[Developed  by  Researcher] 

Table  1 1 .  Selected  CAMIN  Metrics 
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For  requirements  based  in  policy,  regulation,  or  law,  and  for  other 
customer  requirements  the  PM  determines  the  validity  based  on  the  source.  It  is 
important  to  determine  if  the  requirement  is  required  or  simply  advised,  and  under  what 
circumstances  the  requirement  is  valid.  When  a  paying  customer  submits  a  requirement 
with  funding,  the  PM  treats  this  requirement  as  valid  and  relatively  urgent.  The  PM 
treats  requirements  introduced  in  direct  support  of  the  system  mission  to  be  valid  and 
evaluated  based  on  priority. 

For  systematic  maintenance  requirements  of  the  CAMEN  system,  the 
validation  of  the  activities  has  a  basis  in  historical  data.  Experience  has  provided  a 
justification  for  the  baseline  maintenance  activities  that  ensure  consistent  operation  of  the 
system.  Experience  also  shows  probable  consequences  if  there  is  a  lapse  in  maintenance. 

Measure:  Measure  each  funding  driver  for  the  priority/severity  of  need 
and  for  the  likelihood  of  funding  acceptance.  Examples  of  funding  drivers  are  PCRs, 
upgrades  to  hardware  and  software,  and  software/hardware  maintenance  costs. 

Method  of  Calculation:  Collect  input  from  individuals  that  can 
adequately  represent  the  user  community  and  the  mission  requirements  to  evaluate  the 
need.  Measure  the  need  on  a  predetermined  scale  (e.g.  1  to  10),  to  establish  a  baseline 
measurement.  Evaluate  the  time  and  cost  of  implementing  the  new  requirement.  For  the 
CAMIN  program,  the  requirement  baseline  exists  in  terms  of  function  point  estimates. 
The  combination  of  these  measures  provides  the  data  needed  to  adequately  validate  the 
need.  Knowing  the  cost,  the  need,  and  the  benefit  to  the  mission  is  sufficient  to  come  to  a 
business  decision,  based  on  the  resources  available.  For  example,  the  software/hardware 


82 


maintenance  costs  typically  have  a  very  high  need  value  (10).  If  allowed  to  lapse  for  an 
amount  of  time,  the  cost  would  go  back  up  to  the  initial  purchase  price.  The  benefit  to 
the  mission  is  assessed  based  on  the  potential  situation  if  the  need  were  not  met.  Without 
maintenance  agreements,  the  program  experiences  high  risk  of  not  fimctioning,  thus 
becoming  incompatible  with  the  external  environment.  The  cost  is  estimated  based  on 
the  purchase  price  fi-om  the  provider  and  the  costs  incurred  by  the  contractor  in  the 
process. 

b.  Communication  of  Funding  Justification 

Another  key  element  is  to  assess  how  effectively  the  PMO  communicates 
its  justification  of  the  fiinding  needs  to  fimd  managers.  This  research  identified  no 
standard  metric  for  communicating  fimding  needs  for  software  or  for  hardware. 
Therefore,  the  methods  of  measure  suggested  here  are  new,  and  require  evaluation. 
Measures  of  communication  effectiveness  should  address  communicating  the  right  ideas 
about  the  funding  requirements  to  the  right  people. 

Measurement  in  this  area  requires  tools  to  assess  how  well  the  fund 
managers  imderstand  the  funding  intent  and  associated  issues.  The  funding  allocated 
may  not  be  a  useful  measure  of  this,  because  other  issues  arise,  such  as  availability  of 
funding  or  emergency  programs,  which  take  precedence  over  the  need  for  CAMIN. 
Nevertheless,  a  measure  of  funding  requested  vs.  funding  received  provides  a  basis  for 
the  PM  to  study  allocation  of  priorities  and  availability  of  funding. 

A  valid  measure  may  be  available  through  direct  feedback.  The  PM 
should  continue  to  talk  to  fund  managers  throughout  the  funding  process,  and  make 
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appropriate  changes  to  the  communication  process,  based  on  feedback  received. 
Documentation  of  the  communication  process  can  aid  in  future  communication  of 
funding  needs. 

Measure:  Measure  the  degree  of  understanding  by  the  target  audience. 

Method  of  Calculation:  Collect  data  from  fund  managers  and  peers  on 
funding  rates.  Correlate  the  amount  of  funding  requested  to  amount  allocated.  Evaluate 
data  against  other  programs  in  the  same  funding  pool.  If  the  program  fares  unfavorably, 
pursue  action  to  improve  communication.  Solicit  periodic  feedback  from  fund  managers 
to  assess  their  imderstanding  of  program  funding  requirements.  Institute  continuous 
improvement  to  mitigate  problems  when  identified. 

c.  Receipt  of  Required  Funding 

Managing  the  program  requires  sufficient  fimding  to  accomplish  the 
activities  required  in  support  of  the  program.  The  program  budget  provides 
documentation  of  the  funding  requirements.  In  addition,  the  PM  has  flexibility  to 
reallocate  fimding  within  the  program  dollars  to  compensate  for  funding  shortfalls. 
However,  the  PM  must  periodically  assess  the  adequacy  of  the  fimding  in  each  area  to 
use  in  planning  for  future  years.  One  of  the  primary  problems  with  reduced  fimding  in  a 
given  year  is  that  higher  fimding  is  required  in  the  following  year,  to  compensate.  The 
next  year,  delayed  maintenance  causes  higher  maintenance  costs,  and  changes  to 
requirements  become  greater  in  magnitude,  and  therefore  more  expensive.  Documenting 
the  levels  of  fimding,  and  the  effects  of  shortfalls  on  the  requirements  provides  a  record 
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that  supports  future  funding  needs.  The  documentation  also  creates  a  history  to  justify 
limitations  in  meeting  system  requirements. 

Measure:  Measure  and  track  the  adequacy  of  funding  in  each  area. 

Method  of  Calculation:  Collect  data  on  each  new  requirement  and  the 
extent  of  funding  needed  to  meet  that  requirement.  Note  requirements  not  met  for  that 
year,  and  the  effect  on  the  current  year.  Include  an  assessment  of  the  impact  to  the 
program  in  future  years.  In  addition,  document  the  higher  maintenance  cost  each  year, 
and  jumps  in  technology  caused  by  short  funding  in  prior  years. 

2.  System  Availability 

System  availability  is  one  of  the  fundamental  system  requirements.  Lapses  in 
system  availability  over  the  fielded  lifetime  of  the  system  highlight  the  need  for  measures 
in  this  area.  System  availability  encompasses  user  access  to  the  system  or  data  when 
needed. 

a.  Power  to  the  System 

As  discussed  in  Chapter  II,  the  constant  supply  of  power  to  the  server  is 
not  reliable.  The  CAMIN  server  is  dependent  on  the  infiiastructure  within  the  installation 
for  its  power.  The  mission  requirement  for  system  availability  on  a  24-7  basis  is  above 
the  availability  requirement  for  other  computer  systems  on  the  installation.  The  onus  is, 
therefore,  on  the  PM  for  CAMIN  to  take  the  necessary  action  to  meet  its  unique  power 
requirements. 

Before  taking  action  to  prevent  a  power  problem,  the  PM  must  collect  data 
to  determine  the  most  cost-effective  solution.  One  area  to  measure  is  the  extent  and 
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nature  of  power  problems.  The  PM  needs  to  measure  the  occurrence  of  power  outages, 
the  cause  of  those  outages,  and  the  frequency  and  duration.  As  the  PM  takes  action  to 
eliminate  or  mitigate  power  issues,  documenting  the  trends  is  useful  in  justifying  the 
actions  taken. 

Another  area  to  measure  is  the  effectiveness  of  power  protection  systems 
that  are  currently  in  place.  The  systems  to  be  tested  include  uninterruptible  power 
supplies,  the  hot  backup  system  associated  with  the  high  availability  server,  and  the 
power  backups  used  on  the  installation.  The  data  collected  from  this  analysis  provides 
additional  justification  for  enhancing  the  systems,  if  necessary. 

Another  contributing  factor  to  the  importance  of  power  interruptions,  as  a 
valid  measurement  tool,  are  impacts  to  the  user  community.  For  each  power  outage, 
users  should  be  queried  to  determine  the  mission  delay  or  failure  during  that  period. 
Then,  measure  system  usage  rates  over  time  to  determine  potential  deleterious  impacts 
from  a  theoretical  power  outage. 

Measure;  Measure  the  reduction  in  system  availability  due  to  power 

failures. 


Method  of  Calculation;  Keep  track  of  the  cause,  the  frequency,  and 
duration  of  power  outages.  These  data  are  available  from  the  installation  Department  of 
Public  Works.  Query  users  about  specific  impacts  to  mission  caused  by  power  outages. 
Measure  system  usage  over  time  through  reports  from  the  server. 
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b.  Connectivity 

Connectivity  problems  can  reduce  user  access  to  the  system  and  data,  as 
well.  The  PM  is  often  unaware  of  connectivity  issues  imtil  users  identify  problems 
because  of  inability  to  reach  the  C  AMIN  server.  This  forces  the  PM  into  a  reactive  rather 
than  proactive  mode.  Maintaining  connectivity  may  require  maintenance  modification  of 
the  system  to  remain  compatible  with  the  environment  and  network  systems.  Measures 
of  connectivity  involve  three  primary  issues:  1)  network,  2)  firewall,  and  3)  security. 

The  network  is  not  available  for  significant  periods  due  to  local  router 
problems,  virus  protective  actions,  and  other  reasons.  Network  connectivity  is  not  critical 
to  system  operation,  because  the  CAMIN  system  allows  connectivity  through  either 
network  or  phone  connection.  However,  the  phone  connection  is  relatively  slow, 
compared  to  the  network,  inconveniencing  users  and  delaying  schedules.  In  addition, 
there  are  local  policies  that  prevent  phone  connectivity  via  modem,  and  connection  to  the 
local  network  on  the  same  workstation.  The  PM  needs  to  track  losses  of  network 
availability,  and  the  causes,  to  identify  systemic  problems.  The  data  is  available  through 
the  local  network  administrator. 

The  increased  use  of  firewalls  creates  frequent  connectivity  problems. 
One  or  more  firewalls  exist  at  almost  every  CAMIN  location,  and  each  firewall 
configuration  must  allow  CAMIN  client-server  communication.  Network  administrators 
frequently  change  firewall  settings.  A  phone  connection  between  client  and  server 
permits  bypass  of  the  firewall,  if  allowed.  The  PMO  must  remain  in  contact  with  local 
network  administrators  to  track  firewall  configurations  to  assist  when  changes  occur. 
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The  firewall  is  only  one  of  many  security  precautions  that  impede 
connectivity.  Security  is  of  primary  concern  to  network  administrators  today.  Other 
security  precautions  that  impede  connectivity  include  preventing  ping  and  blocking 
access  to  specific  user  segments.  The  primary  organization  within  the  Army  that 
manages  these  security  issues  within  the  military  network  is  the  Theater  Network 
Operations  and  Seciuity  Center  (http://www.conus-tnosc.army.mil/).  a  part  of  the  United 
States  Army  Signal  Command  that  is  located  at  Ft.  Huachuca,  Arizona.  The  PM  must 
collect  and  track  information  on  current  security  activities  to  assure  constant  connectivity 
between  workstations  and  the  CAMIN  server.  For  all  of  these  connectivity  issues,  the 
primary  goal  of  measurement  is  to  be  prepared  to  proactively  manage  problems. 

Measure:  Measure  the  consistency  and  effectiveness  of  coiuiectivity. 

Method  of  Calculation:  Examples  of  the  measures  for  this  aspect  include 
monitoring  and  collecting  data  on  firewalls,  networks,  and  security  changes.  The  PM 
should  assess  trends  to  identify  changes  before  they  occur.  Data  sources  include  all  local 
network  administrators  and  the  U.S.  Army  Theater  Network  Operations  and  Security 
Center.  Data  output  should  be  stored  in  a  central  database  to  identify  trends. 

c.  Workstation/User  Accessibility 

CAMIN  functionality  is  dependent  on  the  ability  of  remote  users  to  access 
the  CAMIN  server  via  the  workstation.  The  previous  section  addresses  network  issues. 
Other  issues  that  provide  system  access  are  proper  configuration  of  the  workstation  and 
correct  user  permissions. 
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As  Chapter  II  discusses,  the  workstation  configuration  is  vital  to  CAMIN 
Client-Server  functionality.  Workstation  configuration  is  difficult  to  assess  and  control  at 
remote  workstation  sites.  Published  policies  are  limited  in  effectively  managing  the 
situation.  However,  the  PM  needs  a  method  to  assure  that  the  workstation  that  is 
connecting  to  the  CAMIN  server  is  the  correct  configuration.  The  method  currently  used 
to  meet  this  requirement  addresses  three  aspects  of  workstation  configuration. 

First,  the  PMO  purchases,  configures,  and  delivers  the  CAMIN 
workstations  to  user  locations.  This  action  provides  initial  control  of  configuration,  but 
has  limited  use  in  guaranteeing  future  control.  There  is  nothing  to  prevent  users  fi-om 
loading  all  CAMIN  custom  and  COTS  software  onto  a  different  workstation. 

Secondly,  the  PMO  must  guarantee  that  only  PM-procured  workstations 
connect  to  CAMIN.  Version  8.0  of  CAMIN  software  prevents  connection  to  the  server 
unless  the  identifying  number  of  the  workstation  network  card  resides  in  the  database. 
This  action  has  been  effective  in  maintaining  control  of  workstations. 

Finally,  the  PM  requires  a  method  of  obtaining  data  from  remote 
workstations  to  assure  compliance.  The  PM  is  currently  implementing  a  system  to  collect 
workstation  configuration  data  at  remote  sites.  This  new  system  provides  users  and 
workstation  administrators  with  the  capability  to  run  a  program,  with  output  containing 
all  required  workstation  parameters.  The  PMO  can  then  request  that  the  remote  location 
run  the  program  at  selected  intervals,  and  transmit  the  output  data  to  the  PMO  for 
collection  and  analysis. 
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In  addition,  accessibility  issue  is  user  permissions.  Users  need  to  have  all 
permissions  that  are  required  to  perform  needed  work.  CAMIN  permissions  restrict  or 
permit  access,  by  organization,  by  installation  and  facility,  by  application,  and  by  action. 
The  user  administrator,  currently  residing  in  the  PMO,  sets  these  permissions.  However, 
the  need  for  specific  permissions  changes  over  time.  The  measure  of  permission  against 
usage  is  key  to  establishing  the  correctness  of  these  permissions  on  an  ongoing  basis. 
These  data  are  available  from  the  CAMIN  server,  and  should  be  measured  and  evaluated 
for  reallocation  of  permissions,  quarterly. 

Measure:  Measure  the  ability  of  users  to  access  the  CAMIN  Server. 

Method  of  Calculation:  The  PM  needs  to  collect  and  evaluate  data  on  the 
configuration  of  CAMIN  workstations  to  identify  trends.  The  PM  is  currently  developing 
a  system  to  obtain  these  data.  Where  incompatibilities  with  the  standard  exist,  the  PM 
must  enforce  correction.  The  PM  also  needs  to  evaluate  user  permissions  against  needs 
and  compare  those  permissions  to  actual  system  usage.  This  data  is  available  on  the 
CAMIN  server.  If  users  are  found  to  have  too  many  or  too  few  permissions,  the  PM  must 
take  corrective  action. 

d.  System  Maintenance 

System  maintenance  employs  preventative  actions  to  keep  the  system 
operational.  Measuring  system  maintenance  involves  measuring  the  frequency,  nature, 
and  impacts  of  maintenance  activities,  and  measuring  the  effectiveness  of  these  activities 
(i.e.  how  often  the  system  in  unavailable  due  to  lack  of  maintenance).  Both  sets  of  data 
are  needed  to  conduct  a  complete  trend  analysis. 
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System  maintenance  activities  include  daily  System  Administration 
activities  and  implementation  of  system  (software,  hardware,  and  other  equipment) 
upgrades.  These  actions  must  occur  in  a  responsible  manner,  ensuring  minimal  impact  to 
the  user,  while  maintaining  consistent  system  operations. 

Daily  System  Administration  activities  include  management  of  file  sizes, 
creating  system  and  data  backups,  managing  accounts  and  passwords,  and  performing 
periodic  system  testing  to  identify  and  correct  problems.  A  daily  log  maintains 
documentation  of  these  activities  to  ensure  compliance  and  identify  trends. 

Implementation  of  system  upgrades  can  involve  taking  the  system  down 
for  an  extended  period  of  up  to  a  week.  It  is  important  to  thoroughly  test  an  upgrade 
before  implementing  it  in  a  live  user  environment.  The  preliminary  testing  can  anticipate 
most  problems  to  precipitate  preventative  action,  but  unforeseen  problems  sometimes 
occur  once  the  upgrade  is  in  an  operational  environment. 

Taking  the  system  down  requires  scheduling  and  coordination  with  system 
users  and  other  customers.  The  PMO  must  build  in  flexibility  to  the  schedule  in  case  a 
short-notice  treaty  inspection  occurs  during  planned  down  time. 

Measure;  Measure  the  frequency,  nature,  and  impacts  of  maintenance 

activities. 

Method  of  Calculation;  The  PM  should  measure  all  aspects  of  proactive 
and  reactive  maintenance  activities  to  determine  trends.  The  individuals  that  perform 
maintenance  must  log  actions  into  a  central  database,  and  take  corrective  action  if 
maintenance  is  not  in  accordance  with  prescribed  levels.  The  PM  should  also  track 
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events  when  the  system  in  imavailable  due  to  a  lack  of  maintenance.  The  event-related 
data  should  include  the  frequency,  duration,  cause,  and  impact  of  the  lapse.  This  data  is 
available  from  system  logs  located  on  the  central  server.  In  cases  where  the  system  is 
unavailable,  the  PM  should  take  corrective  action.  Systemic  causes  of  failure  warrant 
further  action  that  may  require  changing  the  program  strategy. 

3.  Data  and  Output  Accuracy 

As  discussed  in  Chapter  II,  data  and  output  accuracy  are  critical  to  the  success  of 
the  CAMIN  system.  In  order  to  achieve  data  and  output  accuracy,  users  must  possess 
skill  and  knowledge  when  executing  data  input  and  output  choices.  The  PM  provides 

tools  to  prepare  users  and  protect  data.  The  measures  assess  the  effectiveness  of  the 
tools. 

a.  User  Competency  and  Preparedness 

Measuring  user  competence  is  a  political  issue,  tied  to  employee 
performance  ratings  and  user  support  of  the  system.  Direct  assessment  of  user 
competence  creates  distrust  and  contention.  As  stated  in  Chapter  II,  CAMIN  system 
users  are  sometimes  not  computer  literate,  and  CAMIN  data  processing  is  often  an  extra 
duty  included  in  a  workday  of  disparate  activities.  The  CAMIN  PM  has  no  direct  control 
over  the  CAMIN  users  and  their  performance  ratings.  To  gain  user  satisfaction  and 
timely  system  use,  the  PM  provides  a  service  that  the  user  can  trust.  The  PM  must 
balance  the  need  for  improved  competency  against  the  need  for  timely  data  inputs. 

Measures  of  user  competency  may  include  testing  of  users  and/or 
punishment  for  user  errors.  However,  the  impracticality  of  using  these  direct  measures 
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stems  from  the  ability  to  enforce  these  measures,  and  from  the  resulting  customer 
perceptions,  should  these  measures  be  implemented. 

In  addition,  measures  are  not  practical  in  all  cases,  as  system  users  range 
from  those  that  perform  a  wide  range  of  user  activities  to  those  who  only  perform  one  or 
two  basic  actions.  The  test  would  have  to  be  adapted  for  each  user,  eliminating  the 
perceived  “fairness”  of  equality  of  the  measure.  In  other  cases,  testing  may  not  be  a  good 
indicator  for  infrequent  users.  In  the  case  of  infrequent  use,  a  call  to  the  help  line  is  the 
preferable  way  to  provide  instructions  that  link  users  back  to  their  initial  training. 

From  the  PM  perspective,  enforcement  of  a  competency  requirement 
means  that  those  users  that  do  not  pass  testing  are  removed  from  the  user  group.  This 
action  would  be  in  direct  conflict  with  the  need  for  timely  data  input  into  the  CAMIN 
system.  Each  site  is  short  of  staff,  and  there  are  CAMIN  sites  have  no  users  that  could 
pass  the  rigorous  testing. 

Measuring  and  judging  users  has  a  significant  impact  on  customer 
perceptions  and  associated  trust.  There  is  a  concern  that  users,  concerned  for  their  job 
security,  may  hide  data  errors  they  created  in  the  system  to  prevent  punitive  action. 
Another  concern  is  that,  rather  than  make  a  mistake,  the  user  may  do  nothing,  resulting  in 
system  data  that  becomes  grossly  out  of  date.  The  PM  needs  to  maintain  a  relationship  of 
trust  with  the  users,  and  this  type  of  judgment,  on  the  part  of  the  PM,  creates  divisiveness 
and  contention. 

Rather  than  measuring  and  enforcing  strict  guidelines  for  user 
competence,  the  PM  offers  a  selection  of  tools  that  the  user  can  employ  to  improve 


93 


competency.  These  tools  include  training,  access  to  help  line  support,  and  user  manuals. 
Measuring  the  use  of  these  tools  does  not  measure  user  competence.  However, 
measuring  the  frequency  of  use  and  the  perceived  benefits  of  use  provides  a  measure  of 
the  tool  and  its  utility  to  improve  user  competency  and  preparedness. 

Training  for  CAMIN  users  enhances  data  and  output  accuracy.  Training 
provides  instructions  on  system  use,  to  include  how  to  use  the  workstation,  how  to  use 
applications,  and  the  significance  of  data.  Training  also  provides  tips  and  exercises  to 
improve  the  quality  of  learning.  Users  require  retraining  when  the  application  changes 

significantly  and  when  use  of  the  system  has  lapsed  for  only  a  short  period,  typically  one 
month. 

The  help  line  provides  special  instructions  and  other  help  regarding  system 
use  and  problems.  The  help  line  is  a  venue  where  users  present  problems  about  system 
function,  use,  and  data.  As  a  result,  the  help  provided  may  include  instructions, 
intervention,  and/or  PCR  for  system  modification.  Also,  user  manuals  provide  systematic 
instructions  and  diagrams  for  system  users.  Data  and  output  accuracy  can  improve  if  the 
manuals  are  easy  to  use,  informative,  and  factual. 

In  addition  to  measuring  the  tools  provided  to  assist  users,  the  PM  should 
also  keep  track  of  system  users  and  their  abilities.  Consolidation  of  the  data  collected  for 
training  and  through  the  help  line,  data  from  user  interventions,  and  data  indicating 
frequency  and  duration  of  use  by  application,  provide  a  profile  of  each  user  over  time. 
This  profile  may  be  useful  in  establishing  a  baseline  for  competency,  and  in  measuring 
improvement  against  that  baseline. 
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Measure:  Measure  PMO  actions  to  assist  user  competency. 

Method  of  Calculation:  Direct  measures  of  competence  are  beyond  the 
CAMIN  program  control,  but  the  PM  does  control  the  offering  of  materials  and  services 
to  enhance  user  competence.  A  source  of  indirect  measures  is  the  assessment  of  training, 
the  help  line,  and  user  manuals.  Assessment  includes  how  the  PM  provides  these 
services,  how  users  use  the  services,  and  the  results  of  usage.  Additionally,  the  PMO 
should  cormt  training  offerings  and  attendance,  and  have  instructors  provide  an 
assessment  of  trainee  performance  and  improvement.  If  training  is  not  resulting  in  user 
performance  improvement,  the  PMO  needs  to  redesign  the  training  program.  The  PMO 
should  also  measure  the  frequency,  nature,  and  duration  of  help  line  use.  If  trends 
indicate  repeated  problems,  redesign  of  the  system  and/or  manuals  may  be  required. 
Also,  assess  the  quality  of  help  through  trends  in  closeout  rates  and  feedback  from  users. 
This  user  feedback  can  provide  data  to  document  the  benefits  of  the  manuals.  The 
consolidation  of  data  collected  in  this  and  other  metrics  provides  a  measure  of  user 
preparedness  and  an  indicator  of  user  competence. 

b.  CAMIN  Software  Design 

The  CAMIN  software  design  elements  should  concentrate  on  data  and 
output  accuracy.  Three  considerations  should  be  part  of  the  software  design:  1)  ease  of 
use,  2)  data  protections,  and  3)  continuous  improvement.  These  three  elements  provide 
focus  on  the  mission  of  the  system  to  maintain  accurate  data,  in  contrast  to  a  typical  focus 
that  supports  the  continuation  of  the  existing  design. 
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If  the  system  is  easy  to  use,  users  are  more  likely  to  use  the  system,  and 
find  it  easier  to  use  the  system  correctly.  With  a  user  base  that  includes  infrequent  users, 
a  process-based  system  provides  the  benefit  of  reduced  training  requirements  and  more 
accurate  system  data.  The  potential  downsides  of  process-based  applications  are  reduced 
flexibility  for  more  advanced  users,  and  more  time  required  to  perform  a  given  operation, 
(since  users  must  go  through  a  selection  of  all  possible  operations  in  the  process  of 
identifying  the  chosen  one).  Nevertheless,  with  data  accuracy  as  a  primary  goal,  the 
process-based  approach  to  achieve  ease  of  use  is  the  prefeired  alternative.  This  process- 
based  design  change  would  take  place  over  an  extended  period,  through  a  long-term  plan. 
Feedback  from  users  would  indicate  if  the  changes  make  the  system  easier  to  use.  Other 
indicators  of  improvement  are  reductions  in  help  line  calls  and  in  data  errors. 

Another  action  that  improves  ease  of  use  is  the  utilization  of  familiar 
platforms  and  interfaces.  Users  perform  more  comfortably  when  working  in  familiar 
surroundings.  GAMIN  currently  operates  on  a  Windows  ™  NT  Version  4.0  platform. 
When  GAMIN  converted  from  Windows  ™  NT  Version  3.51,  an  improvement  in  the 
comfort  level  of  users  and  local  administrators  was  noted.  Future  changes  being 
considered  for  GAMIN  include  moving  to  a  Windows  ™  2000  platform  and/or  to  a  web- 
based  platform. 

In  addition,  system  data  protections  reduce  the  chance  for  user  error. 
When  systemic  data  errors  are  identifled,  an  investigation  of  causes  may  indicate  that  the 
software  can  be  redesigned  to  prevent  the  error  or  caution  the  user  against  maVing  the 
error.  If  a  redesign  can  help,  the  change  is  documented  in  a  PGR  and  sent  to  the  GGB  for 
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disposition.  Depending  on  the  cost  of  the  change,  the  assessed  benefit,  and  the  binding 
available,  the  change  may  be  included  in  the  next  version  of  CAMIN  software.  The 
measure  of  success  of  the  data  protection  action  is  the  reduced  occurrence  of  the 
identified  data  error. 

A  process  of  continuous  improvement  of  the  design  should  work  toward 
retention  and  improvement  of  data  accuracy.  One  area  of  continuous  improvement  that 
may  enhance  data  accuracy  is  the  leveraging  of  the  system  application  over  several 
programs.  Initially,  the  system  was  designed  to  address  chemical  treaty  requirements 
only.  By  incorporating  the  wholesale  army  accountability  mission,  the  data  is  reassessed 
through  regularly  scheduled  audits  and  inventories.  Additions  of  Materiel  Assessment 
Review  Board  (MARB)  data  storage,  CWPF  accountability  functionality,  and  the  like, 
further  enhance  the  data  accuracy.  Measurement  of  continuous  improvement  is  indirect, 
but  can  be  accomplished  through  reduction  of  data  errors  after  improvements  are 
adopted. 

Measure:  Measure  the  design  characteristics  that  cause  CAMIN  to  meet 

requirements. 

Method  of  Calculation:  Measure  the  ease  of  use  of  the  system  through 
user  feedback.  The  PM  should  use  formal  and  informal  approaches  to  obtain  user 
opinions.  A  measure  of  data  errors  by  frequency,  type,  cause,  and  severity  provides 
benefit,  because  it  addresses  the  design  areas.  However,  this  measure  requires  high 
quality  data  audits,  which  are  outside  PM  control. 
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c.  Data  Audit 

The  data  audit  is  the  most  systematic  method  available  for  assessing  the 
accuracy  of  C  AMIN  data.  Data  audits  must  support  each  type  of  data  within  CAMIN, 
with  representation  from  the  pertinent  mission  areas.  However,  there  are  few  formal 
procedures  in  place  for  audit  of  CAMIN  data.  Table  12  describes  the  organizations 
responsible  for  audit  of  identified  CAMIN  data. 

The  key  organizations  involved  in  data  are  the  NICP,  the  MARB,  the 
treaty  organization,  the  quality  and  maintenance  staff  within  SMT,  and  local  property 
accountability  organizations.  The  NICP  audits  CW  storage  and  destruction  at  the  end  of 
each  destruction  campaign.  The  NICP  also  requires  the  local  custodial  officer  at  the  site 
to  perform  an  annual  inventory  of  all  CW  storage  locations,  reconciled  against  CAMIN 
data. 

When  the  site  receives  notification  of  a  short-notice  treaty  inspection,  the 
local  treaty  compliance  officer  may  conduct  an  emergency  data  review.  This  effort  may 
result  in  surface  corrections,  but  does  not  replace  the  detailed  review  required  to  assure 
data  is  correct.  During  the  treaty  inspections,  the  international  inspection  team  conducts 
a  more  comprehensive  audit  of  CAMIN  treaty  data.  However,  inspections  are 
undesirable  times  to  discover  data  errors,  requiring  explanations  and  negotiations,  and 
breeding  distrust. 
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1  DATA 

1  CATEGORY 

ORGANIZATION 

DATA  AREAS 

NICP 

Storage,  Movement,  and  Destmction, 
Ownership  Codes 

MARB 

MARB  data 

CW 

Treaty 

Storage,  Movement,  and  Destruction 

Quality  and  Maintenance 

Defect  Codes,  Condition  Codes 

Local  Property 

Accountability 

Storage,  Movement,  and  Destruction, 
Hazardous  Waste  Info, 

Treaty 

Storage  and  Destruction 

Former  CWPF 

Local  Property 

Accountability 

Storage  and  Destruction 

Treaty 

Storage,  Movement,  and  Consumption 

Schedule  1 

Local  Property 

Accountability 

Storage,  Movement,  and  Consumption 

Treaty 

Site  Diagrams,  Process  Flow  Diagrams, 
Installation  and  Facility  Info 

NICP 

Installation  and  Facility  Info 

Site 

Information 

Quality  and  Maintenance 

Planographs,  Grid  Definitions, 

Installation  and  Facility  Info 

Local  Property 

Accoxmtability 

Site  Diagrams,  Planographs,  Process 

Flow  Diagrams,  Installation  and 

Facility  Info 

[Developed  by  Researcher] 

Table  12.  CAMIN  Data  Audit  Responsibilities 


None  of  the  other  audits  listed  in  Table  12  take  place  on  a  systematic 
basis.  This  is  a  matter  of  concern  in  support  of  CAMIN  data  accuracy  and  associated 
measures.  The  PMO  does  not  have  sufficient  staff  to  conduct  these  necessary  audits,  and 
the  organizations  with  cognizance  over  the  current  data  are  better  qualified  to  conduct  the 
audits. 
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Data  managers  need  to  periodically  audit  the  CAMIN  database  for 
accuracy.  These  audits  help  to  identify  unique  systemic  data  problems.  The  data 
managers  need  to  have  an  understanding  of  the  data  and  output  requirements.  The 
CAMIN  program  needs  data  managers  from  both  the  Treaty  and  the  wholesale 
accountability  functions  to  work  together  in  the  interest  of  maintaining  consistent  data. 
Evaluation  of  data  errors  can  help  to  determine  if  a  design  change  can  prevent  future 
errors.  Audit  of  data  also  provides  a  method  for  measuring  the  quality  of  system, 
training,  and  other  user  help  systems. 

Measure:  Measure  the  frequency  and  quality  of  audits  to  determine  data 

accuracy. 

Method  of  Calculation:  The  PM  should  measure  the  frequency  and 
output  of  audits,  collecting  data  containing  users,  auditors,  locations,  and  expected 
results.  These  data  are  available  through  queries  of  the  organizations  that  should  be 
conducting  the  audits.  The  PM  does  not  have  control  over  the  audit  process,  but  may 
provide  feedback  of  this  measure  through  the  chain  of  command. 

d.  Data  Interventions 

Through  data  interventions,  the  Data  Administrator  corrects  user  errors, 
intercedes  to  bypass  software  design,  and  performs  mass  changes  to  system  data.  The 
actions  are  performed  through  manipulation  of  the  data  in  the  CAMIN  database.  Data 
interventions  can  be  time  consuming  and  add  risk  to  data  accuracy.  The  action  does  not 
leave  an  audit  trail,  so  documentation  of  the  intervention  is  important.  The  process  for  a 
data  intervention  includes  design  of  the  intervention,  testing  the  intervention  in  the 
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training  database,  and  implementation  and  notification  of  the  users  involved.  A  Data 
Administrator  on  the  contractor  staff  performs  all  data  interventions.  The  Data 
Administrator  only  performs  data  intervention  with  authorization  of  the  PMO. 

It  is  inevitable  that  humans  make  errors.  For  many  important  data 
transactions,  the  CAMIN  system  requires  that  a  second  CAMIN  user  confirm  the 
accmacy  of  the  transaction  through  the  QA  process.  However,  data  errors  continue  to 
occur,  even  with  QA.  Before  automation,  when  users  made  errors,  they  simply  marked 
up  the  paper  document.  In  the  TCM,  the  system  that  immediately  preceded  CAMIN, 
users  could  enter  the  database  and  correct  the  basic  data.  Many  errors  in  the  database 
were  perpetuated  through  these  “back  door”  approaches,  and  CAMIN  does  not  permit 
users  to  access  or  manipulate  the  CAMIN  database.  These  data  interventions  for  error 
correction  improve  the  accuracy  of  data.  Consequently,  the  software  design  has  controls 
to  prevent  users  from  violating  laws,  regulations,  or  treaty,  and  to  protect  the  CAMIN 
data.  In  rare  cases,  there  are  exceptions  to  these  rules,  and  the  Data  Administrator  must 
intercede  to  bypass  the  software  design. 

Infrequently,  a  change  to  policy,  laws,  regulations,  or  treaty  requires  a 
systemic  modification  to  the  CAMIN  data.  For  these  systemic  data  changes,  the  PM 
must  determine  if  it  is  advantageous  to  the  program  for  the  Data  Administrator  to  perform 
a  mass  change  to  the  database,  or  if  users  must  perform  all  transactions  individually. 
Mass  changes  can  improve  system  data  through  consistency  of  data  input  and  through 
customer  satisfaction. 
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Measure;  Measure  the  frequency,  nature,  method,  and  time  spent  on  data 

interventions. 

Method  of  Calculation:  In  all  of  these  cases,  the  Data  Administrator 
should  document  the  process  for  each  intervention  so  that  it  can  be  repeated.  The  Data 
Administrator  should  also  record  the  frequency,  nature,  method,  and  time  spent  on  data 
interventions.  The  PM  can  use  the  data  to  identify  trends  that  may  justify  changes  to  the 
system  design. 

4.  Experienced  IT  Support 

The  need  for  experienced  IT  support  is  fundamental  to  a  successful  program. 
Experience  is  important  for  two  reasons:  1)  The  learning  curves  are  very  long  for  IT 
systems,  and  2)  Retention  is  difficult  for  IT  professionals  in  the  current  economic 
environment,  where  demand  for  technical  ability  is  high.  The  high  performers  assure 
career  success  by  demanding  high  salaries  and  maintaining  leading  edge  skills. 

This  issue  applies  to  both  the  Government  and  contractor  staffs.  Government 
personnel  and  salary  systems  provide  challenges  in  hiring  and  maintainino  these  IT 
professionals.  One  reason  is  that  the  Government  contracting  system  is  oriented  toward 
competition  and  controlling  salary  rates.  As  a  result,  the  PM  for  CAMIN  has  no  IT 
professionals  on  its  staff.  This  creates  a  greater  dependency  on  contractor  IT  experience 
to  maintain  the  system.  Introducing  a  competitive  contract  under  the  current  situation 
would  be  extremely  risky  to  the  program.  This  also  creates  difficulty  in  the  period 
between  contracts,  with  no  Government  staff  to  provide  continuity  for  maintenance. 
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a. 


Leveling  of  Tasks 


The  IT  tasks  for  the  CAMIN  system  consist  of  system  administration,  data 
administration,  design,  development,  and  test.  This  varied  combination  of  IT  work 
requires  knowledge,  skills,  and  abilities  that  rarely  exist  in  a  single  person.  Programs 
invest  time  and  dollars  in  training  and  providing  experience  to  a  group  of  IT 
professionals.  The  program  must  have  the  funding  and  workload  to  support  the  retention 
of  the  group  to  perform  the  IT  functions  over  a  length  of  time  that  is  sufficient  to  gain 
benefit  from  the  investment.  The  minimum  retention  goal  is  three  years.  A  reasonably 
constant  amount  of  work  in  each  of  these  areas  supports  the  retention  of  the  skilled  key 
IT  personnel.  Level  funding  is  one  key  element  in  maintaining  constant  workload. 
Another  element  that  can  assist  fi-om  a  contract  perspective  is  the  type  of  contract. 

Level  fimding  can  occur  through  effective  budgeting  and  responsible 
management.  The  type  of  contract  and  method  of  administration  can  improve  the  PM’s 
ability  to  retain  an  experienced  IT  support  staff.  The  contract  type  should  provide 
consistent  and  adequate  funding,  and  work  levels.  Multiyear  contracting  can  be  helpful 
in  leveling  the  tasks.  The  PM  needs  to  periodically  assess  funding  and  contracting  issues 
to  determine  the  adequacy  in  maintaining  a  stable  IT  workforce.  This  data  is  available 
within  the  PMO  and  through  contractor  reports. 

The  IT  Fund,  defined  in  U.S.  Code,  Title  40,  Section  757,  (Ref  21) 
provides  valuable  tools  in  addressing  these  IT  issues  for  the  contract.  The  Government 
Services  Administration  has  authority  to  award  and  administer  IT  multi-agency  contracts 
and  to  fund  contracts  on  a  reimbursable  basis  through  a  separate  IT  fimd.  GSA  is  the 
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only  Executive  Agent  of  that  fund,  designated  by  0MB  under  the  Clinger  Cohen  Act 
(Ref.  5).  In  order  to  promote  the  efficient  management,  coordination,  operation,  and 
utilization  of  resources;  the  fund  has  been  established  without  fiscal  year  limitation,  and 
allows  for  the  use  of  multi-year  contracts  for  IT  hardware,  software,  and  services.  This 
multiyear  contracting  authority  can  be  used  when  the  following  criteria  are  met;  funds  are 
available  and  adequate,  the  contract  is  fully  competitive,  the  need  continues  for  the 
contract  period,  it  yields  substantial  cost  savings,  and  it  cannot  exclude  small  businesses. 

The  IT  Fund  provides  very  valuable  tools,  which  can  be  used  when  contracting  through 
GSA. 

Measure:  Measure  the  amount  and  types  of  work  under  the  contract  and 
the  contract  vehicle. 

Method  of  Calculation;  Periodically  assess  if  funding  and  contracting  are 
adequate  to  maintain  a  stable  IT  workforce.  This  data  is  available  within  the  PMO  and 
through  contractor  reports.  If  work  is  inadequate,  the  PM  should  provide  data  to  decision 
makers  to  help  make  informed  decisions  about  the  program  and  funding. 

b.  Challenges  and  State  of  the  Art 

There  are  many  good  reasons  for  a  program  to  keep  up  with  state  of  the  art 
technology,  and  maintaining  experienced  IT  professionals  on  staff  may  be  the  least 
important  of  those  reasons.  However,  keeping  a  challenging,  “state  of  the  art”  program 
provides  the  program  developers  and  other  IT  professions  with  another  reason  to  remain 
on  the  program,  as  it  provides  a  continuous  learning  environment.  The  technology  on  the 
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program  should  not  stagnate  to  the  point  where  schools  and  the  rest  of  the  IT  world  no 
longer  use  that  technology. 

Measure:  Measure  the  trends  in  technology. 

Method  of  Calculation:  Assess  the  CAMIN  system  based  on  these  trends. 
Obtain  information  on  technology  trends  through  trade  journals,  conferences,  and 
contacts.  Establish  program  strategies  based  on  sound  business  practices,  using  these 
trends  to  support  decisions. 

5.  Requirements  Changes 

Changes  to  system  requirements  are  perpetual  through  the  life  of  the  system.  The 
changes  provide  the  continuation  of  system  functionality  in  a  changing  environment. 
Frequency  of  changes  drives  the  pace  of  the  system  and  funding  requirements. 

a.  Changes  to  External  Environment 

Changes  to  COTS  software  and  hardware  that  are  part  of  the  CAMIN 
system  drive  changes  to  CAMIN.  The  reasons  to  adopt  these  COTS-driven  requirements 
changes,  include  maintenance  support  availability  and  functionality  within  the  external 
policies  and  environment.  These  factors  are  out  of  the  control  of  the  PM.  The  cost 
effectiveness  of  COTS  is  realized  in  the  avoidance  of  development  costs  and  the 
avoidance  of  maintenance  personnel.  Despite  these  factors,  the  Government  and 
commercial  industry  have  accepted  this  trade-off,  and  lise  COTS  as  much  as  possible,  as 
general  policy. 

However,  maintenance  support  is  only  available  as  long  as  the  COTS 
developer  chooses  to  support  it.  The  IT  industry  is  so  volatile  that  support  to  a  legacy 
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system  often  stops  after  about  a  year  of  fielding  a  newer  version  of  that  system.  The 
CAMIN  has  flexibility  even  after  software  support  has  stopped.  The  COTS  software  will 
function  as  long  as  the  interfaces  still  work,  but  may  fail  to  comply  with  new  policies. 
Once  support  stops  on  COTS  hardware,  the  manufacturer  does  not  offer  maintenance 
contracts  and  replacements  parts  become  difficult  to  find. 

As  a  result,  the  CAMfM  PMO  must  try  to  remain  compliant  with  the 
fi-equently  changing  external  environment  and  policies.  Recent  changes  that  fall  under 
this  category  are  networks  and  firewalls,  Y2K  compliance,  and  handicapped  accessibility 
to  web  sites.  These  policy  changes  are  often  unpredictable,  implemented  on  short  notice, 
and  imposed  with  no  associated  funding.  The  common  link  is  that  these  external  changes 
are  beyond  the  control  of  the  PM,  and  yet  the  PM  must  adapt  to  them.  Program  planning 
and  fimding  must  allow  for  these  kinds  of  changes.  The  PMO  must  maintflin  awareness 
of  environmental  changes  from  COTS  developers,  and  from  Federal  and  industry  policy 
organizations. 

Measure:  Measure  the  COTS  software,  hardware,  and  the  network 
environment  to  proactively  adjust  to  changes. 

Method  of  Calculation:  Track  the  history  of  changes  to  predict  future 
needs.  The  PM  can  obtain  information  through  Federal  and  trade  journals,  industry 
conferences,  and  contacts.  Establish  program  strategies  based  on  sound  business 
practices,  using  these  data  to  support  decisions. 
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b. 


Evaluation  of  Changes 


The  evaluation  of  changes  to  the  CAMIN  system  takes  place  during  CCB 
meetings.  Voting  membership  in  the  meetings  includes  user  representatives,  funding 
representatives,  and  mission  representatives.  Consideration  of  priority,  complexity  and 
severity,  and  costs  helps  the  CCB  and  PM  make  appropriate  decisions,  given  resource 
limitations.  As  indicated,  one  consideration  in  evaluating  changes  is  priority.  Priorities 
are  pivotal  to  decisions  making,  and  often  determining  if  the  change  is  important  enough 
to  pursue.  A  determination  of  the  type  of  change  (defect,  problem,  or  enhancement)  also 
has  influence  on  priority  designation.  The  designations  of  priority  and  type  are  set  by  the 
user  community  and  by  the  funding  organizations,  with  input  from  the  contractor. 

Fielding  of  changes  to  software  can  occur  either  through  a  full  version 
release  or  through  a  patch.  The  contractor  evaluates  each  change  to  determine  its 
siiitability  to  be  included  in  a  patch.  In  addition,  the  size  of  the  change  determines 
whether  to  field  the  change  through  the  NIPRNET  or  a  compact  disk. 

Another  factor  in  the  decision  is  the  complexity  and  severity  of  changes. 
The  contractor  determines  the  complexity  through  function  point  analysis,  and  provides 
the  data  to  the  CCB.  Managers  may  decide  to  postpone  changes  that  consume  large 
amounts  of  time  and  other  resources,  in  order  to  accomplish  actions  that  are  more 
pressing.  The  cost  factor  for  each  PCR  is  significant,  and  is  determined  from  the  function 
point  analysis,  and  the  consumption  of  resources  may  make  them  prohibitively  costly  and 
time  consuming  to  implement. 
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For  implementation,  the  CCB  collects  all  of  these  data  to  use  in 
determining  the  disposition  of  change  requests.  The  CCB  analysis  results  in  an  action  for 
each  PCR  that  either  approves,  disapproves,  leaves  in  an  “under  consideration”  category, 
withdraws,  or  designates  as  overtaken  by  events.  When  funding  becomes  available,  the 
PM  considers  approved  and  urgent  actions  in  generating  a  contract  modification  to 
implement  a  version  upgrade  to  the  CAMIN  software. 

Measure:  Measure  the  priority,  type,  complexity,  and  severity  of  change 

requests. 

Method  of  Calculation:  For  each  PCR,  determine  priority,  type  of 
change,  suitability  of  the  change  to  be  fielded  in  a  patch,  complexity  and  severity,  and 
cost.  Data  comes  from  the  contractor,  from  CCB  members,  and  from  the  submitter  of  the 
PCR.  Data  is  stored  in  central  database  and  used  in  tracking  requirements  and  making 
upgrade  decisions. 

B.  LESSONS  LEARNED 

The  analysis  from  this  thesis  results  in  lessons  that  can  provide  benefit  to  the 
CAMIN  PM.  The  lessons  may  also  apply  to  other  fielded  software  systems  and  IT 
systems  managers,  as  well.  These  lessons  learned  are  identified  in  the  following 
paragraphs. 
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Ensure  that  decision  makers  fully  understand  the  CAMIN  program 
requirements  (software,  hardware,  people,  dollars). 

Decision  makers  must  have  the  salient  facts  before  making  informed  decisions 
about  the  program.  Those  making  decisions  about  software,  hardware,  people,  and 
dollars  may  not  realize  the  significant  differences  between  IT/  fielded  software  systems 
and  other  programs.  For  example,  decision  makers  need  to  imderstand  that  updates  of 
software  are  more  fi-equent  than  hardware  and  persist  through  the  life  of  the  system. 
Further,  each  individual  program  has  imique  characteristics.  The  PM  must  keep  decision 
makers  informed  on  program  issues  and  changes  to  the  program. 

Ensure  that  users  and  their  supervisors  understand  and  take  responsibility 
for  having  competent  (trained  and  prepared)  CAMIN  system  users. 

Users  are  essential  to  CAMIN  and  its  data,  and  the  users  on  the  CAMIN  system 
experience  unique  challenges  of  insufficient  time  and  technology  skills.  Managers  do  not 
understand  what  is  necessary  in  terms  of  time  and  money  to  select  and  maintain 
competent  system  users.  Beyond  that,  many  cannot  afford  to  expend  the  necessary 
resources  on  training  and  maintenance  of  users.  For  example,  managers  and  users  may 
not  realize  that  when  use  of  the  system  is  infrequent,  users  need  to  spend  additional  time 
on  training  and  practice  to  maintain  skills. 

Managers  need  to  monitor  the  performance  of  CAMIN  users  under  their 
supervision  to  assess  the  quality  of  performance.  Insufficient  audits  take  place  at  the 
oversight  level  to  detect  errors  in  data.  Site  responsibility  for  data  is  very  important  in 
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cases  where  only  the  site  personnel  know  the  “ground  truth”  about  the  data,  and  can 
detect  data  errors. 

Ensure  that  mission  representatives  understand  the  importance  of  data 

audits. 

Data  audits  can  significantly  improve  the  quality  of  data  in  CAMIN,  and  can  help 
to  monitor  the  skill  level  of  system  users.  The  lack  of  sufficient  audits  increases  the 
program  risk,  by  allowing  existing  data  errors  to  go  uncorrected.  Those  in  a  position  to 
audit  CAMIN  data,  as  shown  in  Table  12,  need  to  understand  the  importance  of  audits  to 
the  data  relevant  to  particular  programs.  Performance  of  the  data  audits  requires 
knowledgeable  personnel,  dedicating  considerable  time  and  effort  to  the  action.  The 
required  frequency  of  data  audits  is  variable,  dependent  on  the  volatility  of  data  in  the 
system. 

Maintain  knowledge  of  trends  and  pending  actions  in  the  external  IT 
environment,  for  both  Government  and  industry. 

The  environment  has  a  significant  influence  on  the  CAMIN  program  and  its 
resources.  Changing  client  needs  and  changes  to  the  external  environmental  drive 
requirements  changes  and  subsequent  system  updates.  External  requirements  for 
CAMIN  can  come  from  many  sources,  including  the  command  CIO,  AMC,  Army,  DoD, 
Congress,  and  the  Treaty  organizations.  Changes  come  in  the  form  of  policy,  regulation, 
or  law,  and  are  rarely  accompanied  with  the  funding  required  to  implement. 
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Work  toward  Thin  Client  Architecture  for  CAMIN. 


For  systems  that  have  remote  users.  Thin  Client  is  often  more  practical  than 
Client-Server.  The  Case  Interviews  revealed  a  consistent  conclusion  with  the  Thin  Client 
design.  For  the  CAMIN  system,  each  step  toward  total  thin  client,  or  web-based, 
architecture  reduces  the  number  of  CAMIN  workstations  the  program  has  to  procure  and 
support.  Firewall  issues  become  minimal.  Access  to  data  is  greatly  improved,  as  users 
with  a  CAMIN  password  can  log  into  CAMIN  from  workstations  with  Internet 
connectivity.  This  action  is  consistent  with  the  program  objective  to  conform  to  the 
external  environment. 

Achieve  a  constant  workload  for  the  CAMIN  contractor  to  retain 
experienced  IT  support. 

Experienced  IT  support  helps  a  program  to  perform  more  efficiently.  Since  the 
CAMIN  program  has  no  IT  professionals  in  the  PMO,  there  is  a  strong  dependency  on 
the  contractor  to  provide  continuity  as  well  as  experience.  Performance  becomes  more 
efficient  and  consistent  when  leaning  curves  and  experience  curves  are  minimized.  A 
core  of  experienced  personnel  can  provide  leadership  and  guidance  to  a  staff  of 
developers  working  on  a  version  upgrade  or  patch. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Following  is  a  brief  discussion  of  each  of  the  conclusions  that  arise  from  the 
research  and  analysis.  The  analysis  of  a  specific  program  is  required  to  determine  issues 
and  the  associated  required  measures.  For  example,  the  research  into  possible  metrics 
does  not  reveal  measures  associated  with  communicating  program  funding  needs.  The 
examination  of  system  issues  for  CAMIN  demonstrates  the  clear  need  for  metrics  in  this 
area. 

The  conclusions  that  follow  should  assist  in  providing  statistical  data  to  support 
funding  justification.  The  organizations  that  justify  and  analyze  funding  for  CAMIN 
require  supporting  documentation  for  these  actions,  but  planning  what  data  are  required 
or  requested  in  the  future  is  rarely  possible.  A  PM  benefits  from  developing  methods  of 
easily  capturing  measurements  for  different  aspects  of  the  program.  CAMIN  uses 
databases  to  track  action  items,  CCB  PCR  information,  requirements  baseline  and 
changes,  and  the  like.  However,  additional  metrics  are  necessary  if  the  program  is  to  be 
supportable  in  the  long-term. 

The  CAMIN  program  requires  software  metrics  and  fielded  system  metrics. 

The  analysis  shows  that  the  CAMDDN  PM  must  use  metrics  from  both  software  and 
from  fielded  systems  to  address  primary  programmatic  issues.  Key  areas  not  addressed 
in  software  metrics  are  customer  satisfaction  and  availability.  Both  of  these  areas  are 
essential  to  evaluation  and  justification  of  fielded  systems. 
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The  CAMIN  program  requires  metrics  that  are  specific  to  the  program, 
utilizing  standard  and  non-standard  metrics. 

The  CAMIN  system  needs  metrics  that  are  not  standard  to  software  or  fielded 
systems.  One  example  is  a  metric  to  address  communication  of  funding  requirements. 
The  funding  sources  for  CAMIN  are  inexperienced  in  software  systems,  and  are 
unfamiliar  with  the  iterative  nature  of  PDSS  management. 

The  PMO  must  continually  evaluate  metrics  for  the  life  of  the  system. 

The  program  issues  change  through  the  life  of  a  system.  Political,  policy  and 
requirements  changes  are  continuous,  and  drive  program  issues.  Issue  changes  require  a 
review  of  program  metrics. 

A  model  for  establishing  metrics  that  this  thesis  uses  is  depicted  in  Figure  9.  The 
first  step  is  to  establish  issues,  through  a  thorough  system  analysis,  including  evaluation 
of  the  program’s  internal  and  external  environment.  The  issues  should  also  consider 
program  strengths,  weaknesses,  opportunities,  and  threats.  After  issues  become  clear,  the 
issues  must  be  refined  into  fundamental  areas  of  measurement.  The  metrics  and 
measures  of  each  issue  must  be  established,  along  with  methods  for  data  collection  and 
analysis.  Implementation  and  data  collection  provide  the  data  needed  for  evaluation. 
The  evaluation  process  may  yield  changes  to  system  management,  operations,  or 
measures. 

These  changes  must  be  fed  back  into  the  metrics  process  to  maintain  a  current 
system  of  measures.  Finally,  the  PM  must  reevaluate  the  entire  process  of  establishing 
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the  program  issues  that  drive  metrics.  The  reevaluation  should  occur  annually, 
concurrent  with  revision  to  the  program  strategic  plan. 


Fuo^^eitt 


Use  Feedback  to  Improve  Metrics  and  Measures 


[Developed  by  Researcher] 

Figure  9.  Model  for  Selecting  Program  Metrics 


Management  Information  Systems  are  unique. 


Management  Information  Systems  requirements  change  frequently,  affecting 
overall  program  management  and  funding.  These  systems  manage  financial,  office. 


property,  and  other  office-oriented  missions.  Drivers  for  the  requirements  changes 
include  external  environment,  policy,  user  group,  and  COTS. 

Each  Helded  software  system  may  require  unique  metrics. 

The  metrics  required  for  CAMIN,  at  this  point  in  the  life  cycle,  may  be  different 
from  other  fielded  software  systems.  Each  fielded  software  system  PM  must  study 
program  issues  to  develop  a  set  of  metrics  for  that  particular  program.  Program  managers 
should  not  rely  on  published  standards  for  metrics. 

B.  RECOMMENDATIONS  TO  CAMIN  PROGRAM  MANAGER 

The  CAMIN  PM  should  adopt  the  set  of  metrics  in  Table  1 1  for  managing  the 
CAMIN  program.  These  metrics  are  the  correct  metrics  for  the  system,  as  it  exists  today. 
The  measures  derived  through  application  of  these  metrics  should  reside  in  a  central 
repository  where  all  who  are  involved  in  the  program  management  process  of  CAMIN 
can  access  and  use  the  data  collected. 

The  PMO  should  reassess  these  metrics  and  data  each  year,  using  the  process 
described  in  Figure  9.  Reassessment  should  occur  in  consonance  with  the  publishing  of 
the  Annual  Business  Plan  for  CAMIN.  The  reassessment  should  look  at  the  cost  of 
collecting  the  data,  as  a  factor  in  assessing  the  value  of  the  measure.  The  PM  should 
revise  the  Annual  Business  Plan  to  reflect  the  current  metrics  and  results  to  date.  The  PM 
should  use  the  process  depicted  in  Figure  9  in  the  analysis. 
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The  lessons  learned  from  this  research,  described  in  Chapter  IV  of  this  thesis  and 
listed  here,  become  additional  recommendations  to  the  PMO.  The  PM  should  strive  to 
achieve  these  actions,  within  the  control  and  purview  of  the  organization. 

•  Ensure  that  decision  makers  fully  imderstand  the  CAMIN  program 
requirements  (software,  hardware,  people,  dollars). 

•  Ensure  that  users  and  their  supervisors  understand  and  take  responsibility  for 
having  competent  (trained  and  prepared)  CAMIN  system  users. 

•  Ensure  that  mission  representatives  understand  the  importance  of  data  audits. 

•  Maintain  knowledge  of  trends  and  pending  actions  in  the  external  IT 
environment,  for  both  Government  and  industry. 

•  Work  toward  Thin  Client  Architecture  for  CAMIN. 

•  Achieve  a  constant  workload  for  the  CAMIN  contractor  to  retain  experienced 
IT  support. 

C.  RECOMMENDATIONS  TO  OTHER  PDSS  MANAGERS 

PDSS  managers  who  are  interested  in  defining  appropriate  metrics  for  their 
program  should  follow  the  same  template  as  was  used  in  this  thesis,  as  shown  in  Figure  9. 
Developing  a  list  of  issue  areas  for  the  unique  fielded  software  system  is  the  first  step  in 
this  process.  For  systems  where  issues  are  common  to  CAMIN,  the  analysis  is  complete. 
For  different  issues,  a  review  of  the  candidate  metrics  may  help  to  develop  a  tailored  list 
of  metrics  for  a  fielded  software  system.  During  the  analysis,  PDSS  managers  should 
keep  an  open  mind  to  new  metrics  that  may  fit  their  particular  program  and  its  issues. 
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D.  FUTURE  RESEARCH 

This  section  looks  at  areas  that  warrant  future  research.  The  areas  of  PDSS  and 
metrics  bring  a  wide  range  of  ideas  for  research,  encompassing  the  metrics  for  CAMIN 
program  and  other  programs  in  the  PDSS  phase.  Additionally,  there  is  a  wide  range  of 
other  research  to  be  done  in  the  study  of  managing  fielded  software  systems.  This  area  is 
becoming  more  important  as  more  software  systems  are  fielded. 

1.  Evaluate  Outcome  from  Metrics  Selected  on  CAMIN  Program 

An  area  of  research  that  grows  directly  from  this  thesis  is  assessment  of 
implementing  the  thesis  recommendations.  The  analysis  would  contrast  the  value  of  the 
metrics  against  the  imagined  benefit,  assess  the  measures  against  the  five  issue  areas  for 
the  reporting  period,  and  evaluate  the  cost  of  the  implementing  and  using  these  metrics. 

2.  Evaluate  Selected  Metrics  for  Other  PDSS  Programs 

Another  area  that  relates  to  this  thesis  is  to  assess  how  another  fielded  software 
system  could  use  the  outcome  of  this  thesis.  This  action  would  include  measuring  the 
value  of  the  metrics  against  the  imagined  benefit.  It  may  also  compare  the  results 
identified  for  the  other  fielded  software  systems,  and  compare  to  the  results  on  the 
CAMIN  program. 

3.  Evaluate  Other  Aspects  of  Fielded  Software  Systems 

Metrics  is  only  one  area  of  interest  in  the  program  management  of  fielded 
software  systems.  The  issues  uncovered  during  the  course  of  case  interviews  and  other 
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data  collection  require  further  analysis.  This  is  a  short  list  of  ideas  for  future  research 
into  PDSS  management. 

Research  into  the  nature  and  necessity  of  requirements  changes  after  fielding 
would  be  beneficial.  The  fi-equent  changes  to  requirements  for  fielded  software  drive 
cost  and  workload  for  the  life  of  the  system.  Many  of  the  requirements  changes  for 
CAMEST  are  not  foreseeable,  so  the  results  of  such  analysis  would  be  instructive  for  PMs. 

The  maintenance  costs  for  PDSS  seem  imcommonly  high  to  most  Army 
managers.  The  annual  costs  run  about  20  percent  of  the  total  development  cost  of  a 
system.  A  case  analysis  of  program  management  costs  for  PDSS  Management 
Information  Systems  across  Army  or  DoD  would  provide  useful  baseline  data  and 
analysis. 

Another  area  of  research  evaluates  trends  in  system  architecture  and  management 
that  can  improve  system  access,  ease  of  use,  and  avoid  cost  through  the  system  life  cycle. 
As  with  hardware  systems,  the  initial  design  for  software  systems  can  drive  up  costs  in 
maintenance  and  in  logistics  support.  The  DoD  approach  is  to  maximize  use  of  COTS 
and  standard  programming  languages.  This  research  may  support  the  value  of  the 
recommendations. 

Case  research  into  PDSS  to  compare  management  techniques  may  provide  insight 
into  successful  and  unsuccessful  approaches  within  DoD.  The  size,  complexity,  and 
visibility  of  the  system  play  into  the  management  techniques.  For  example,  a  system 
rated  with  a  high  acquisition  category  requires  a  classic  PM  structure  within  DoD,  and 
has  rigid  requirements  during  the  system  development.  After  fielding,  DoD  regulations 
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are  not  as  directive  of  system  management  requirements  as  those  established  for 
developmental  systems. 

Finally,  the  study  of  retention  of  IT  professionals  on  a  fielded  software  systems 
can  provide  data  and  analysis  that  would  address  both  DoD  employees  and  contractors. 
The  results  may  contribute  to  hiring  practices  and  pay  scale  policies  within  DoD,  and  to 
IT  contracting  practices.  The  practice  of  acquiring  and  retaining  qualified,  capable,  and 
consistent  IT  professionals  is  the  core  of  a  successful  development  and  sustainment 
program. 
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APPENDIX  A.  CORRESPONDENCE 


The  author  requested  these  data  from  the  managers  of  systems  considered  for  the 
case  interviews.  The  author  conducted  Case  Interviews  on  phone  and  in  person. 

Full  Name  of  System 
Type  of  System 
Mission 
Scope 

Size  (SLOCs) 

Architecture  (very  general) 

Number  and  types  of  users  and  connectivity 

Brief  History  of  development  and  versions 

Metrics  and  Measures,  and  other  program  management  Tools 
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APPENDIX  B.  CAMIN  WORKSTATIONS  AND  USERS 
[After  Ref.  14] 


LOCATION 

FACILITY 

FACILITY 

TYPE 

WKSTNS 

USERS 

WEB 

ONLY 

USERS 

Aberdeen,  MD 

Aberdeen  Chemical 
Demilitarization  Facility 
(ABCDF)  (Under  Construction) 

CW  Destruction 

0 

0 

Aberdeen,  MD 

Edgewood  Chemical  Activity 
(ECA)  (Active  Stockpile  Site) 

CW  Storage 

3 

2 

Aberdeen,  MD 

Edgewood  Chemical  Biological 
Center  (Active  Operations) 

Schedule  1 

Single  Small- 
Scale  Facility 
(SSSF) 

2 

3 

Aberdeen,  MD 

Program  Manager  for  Chemical 
Demilitarization  (PMCD) 

Management  and 
Operations 

2 

5 

Aberdeen,  MD 

Soldier  and  Biological  Chemical 
Command  (SBCCOM) 

Management  and 
Operations 

4 

9 

Anniston,  AL 

Anniston  Chemical  Activity 
(ANCA)  (Active  Stockpile  Site) 

CW  Storage 

5 

2 

Anniston,  AL 

Anniston  Chemical 
Demilitarization  Facility 
(ANCDF)  (Under  Construction) 

CW  Destruction 

0 

2 

Commerce  City, 
CO 

Rocky  Mountain  Arsenal 
(Active  Destruction) 

Former  CWPF 

2 

0 

Dugway,  UT 

Dugway  Proving  Ground 
(Active) 

CW  Storage 

2 

2 

Fort  Leonard 
Wood,  MO 

Fort  Leonard  Wood  10  kg 

Facility  (Active  Operations) 

Schedule  1  10  kg 
Facility 

1 

2 

Hawthorne,  NV 

Destruction  Facility 
(Intermittent,  —Currently 

Inactive) 

CW  Destruction 

0 

0 

Johnston  Island 

Johnston  Atoll  Chemical  Agent 
Destruction  System  (JACADS) 
(Active,  almost  complete) 

CW  Destruction 

4 

6 

Lexington,  KY 

Blue  Grass  Chemical  Activity 
(BGCA)  (Active  Stockpile  Site) 

CW  Storage 

4 

0 

Lexington,  KY 

Blue  Grass  Chemical 
Demilitarization  Facility 
(BGCDF)  (Planned) 

CW  Destruction 

0 

0 
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LOCATION 

FACILITY 

FACILITY 

TYPE 

Newport,  IN 

Newport  Chemical 

Demilitarization  Facility  (NECDF) 
(Under  Construction) 

CW  Destruction 

Newport,  IN 

Newport  Chemical  Depot  (NECD) 
(Active  Stockpile  Site) 

CW  Storage 

Ne\vport,  IN 

Newport  Chemical  Depot  (NECD) 
(Active  Destruction) 

Former  CWPF 

Pine  Bluff,  AR 

Pine  Bluff  Chemical  Activity 
(PBCA)  (Active  Stockpile  Site) 

CW  Storage 

Pine  Bluff,  AR 

Pine  Bluff  Chemical  Activity 
(PBCA)  (Active  Destruction) 

Former  CWPF 

Pine  Bluff,  AR 

Pine  Bluff  Chemical 
Demilitarization  Facility  (PBCDF) 
(Under  Construction) 

CW  Destruction 

WKSTNS 

USERS 


Pueblo,  CO 


Pueblo,  CO 


Tooele,  UT 


Tooele,  UT 


Tooele,  UT 


Umatilla,  OR 


Umatilla,  OR 


Washington,  DC 


Facility  (PUCDF)  (Planned) 


Pueblo  Chemical  Depot  (PCD) 
(Active  Stockpile  Site) 


Chemical  Agent  Munitions 
Destruction  System  (CAMDS) 
(Active) _ 


Deseret  Chemical  Depot  (DCD) 
(Active  Stockpile  Site) 


Tooele  Chemical  Demilitarization 
Facility  (TOCDF)  (Active) 


Umatilla  Chemical 
Demilitarization  Facility 
(UMCDF)  (Under  Construction) 


Umatilla  Chemical  Depot  (UMCD) 
(Active  Stockpile  Site) 


Department  of  Army 


CW  Destruction 


CW  Storage 


CW  Destruction 


CW  Storage 


CW  Destruction 


CW  Destruction 


CW  Storage 


Management  Only 


Washington,  DC 

US  Army  Nuclear  and  Chemical 
Agency 

Management  Only 

0 

1 

Washington,  DC 

Defense  Threat  Reduction  Agency 

Management  Only 

0 

3 

Honolulu,  HI 

US  Army  Pacific  (USARPAC) 

Management  Only 

0 

1 

TOTALS 


53 


68 


NT  Applications 


APPENDIX  C.  CAMIN  LINES  OF  CODE 


[From  Ref.  4] 


Version  8  Components 


Ad  Hoc 


Alert  Catalog 


Version  8  Version  7.5  Version  7 


CWPF  Scheduling 
Data  Administration 
Declaration  Catalog 
Declared  Chemical 
Declarations 
Data  Download 
Material  Release 
Notification  Catalog 
Notifications 
Notify  Server 
RAS  Manager 
Reports 
Schedule  1 
Schedule  2, 3,  Other 

Site  Diagrams _ 

Site  Information 
Stock  Records 
User  Administration 
NT  Total 


Annual  Inventory 

CAMIN  DLL 

87361 

CAMIN  Settings 

CAMIN  Setup 

2548 

2641 

Compare 

6063 

6162 

CW  Destruction  Reporting 

0 

9352 

CW  Destruction  Scheduling 

0 

11731 

CW  Items 

CWPF  History 

2i 

23348 

565 


3265 


124 

27 


9274 


2477 


3086 


2487 


29667 


22122 


7952 


16458 


59516 


372603 


24326 


0 


3250 


1  4445 

4687 

1997 

1934 

27873 

22345 

22321 

0 

0 

21111 

21073 

36435 

37437 

9903 

10318 

350869 

346416 

Server  Applications 


Version  8  Components  Version  8  Version  7.5  Version  7 


Chlst 

58: 

52{ 

5  473 

chlst_lib 

1988i 

JEHQE 

9110 

combine 

compresslib 

467f 

469f 

dbupdate 

IHIKS 

IHHKS 

390 

Dcelib 

lEBKEi^ 

4492 

decl_lib 

1041S 

9813 

im 

decl_ipts/decl_cw_dest 

23462 

23458 

declrpts/declcwinitial 

19091 

19091 

decl_rpts/decl_cw_schedl 

0 

7752 

13635 

decl_rpts/decl_cw_sched2_3_doc 

52036 

52034 

21657 

decl_rpts/decl_cwpf_ami_dest_rpt 

5224 

5220 

3333 

decl_ipts/decl_cwpf_ann_dest_sched 

HHH 

16483 

decl_ipts/decl_cwpf_initial 

17002 

decl_rpts/ decl_view_comments 

1162 

1162 

delete_elements 

3212 

1723 

dispatcher 

3615 

3111 

extemal_trans 

0 

0 

2605 

file_server 

eib^ 

1911 

1825 

fwd_n_rej  ect_docs 

5512 

ing_lib 

2090 

2090 

2714 

intemal_trans 

0 

0 

3156 

issue-docs 

1470 

1405 

Lib 

1922 

localdata 

137 

135 

planagraph 

3418 

post_docs 

lE^^I 

q_combine_msg 

qa_process 

10447 

4997 

3254 

Server_lib 

1822 

1448 

sqlbatch 

37613 

12870 

stock_trans 

0 

slashgo 

76 

76 

76 

141 

141 

142 

272 

272 

267 

add_server_users 

324 

324 

354 
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Version  8  Components _ Version  8  Version  7.5  Version  7 


caminarchive 

51 

57 

57 

camin-files-clean 

4063 

4555 

0 

camin_shutdown 

57 

57 

57 

camin_test 

294 

294 

0 

check_backup 

374 

374 

li^MI 

check_backup_fs 

386 

772 

i^Sl 

e 

clr_rtn 

107 

|||||^■|■^ 

dcewho 

262 

262 

68 

W 

del_dce_users 

94 

95 

a 

a 

dirlist 

0 

0 

0 

< 

u 

52 

52 

o 

t 

list_dce_users 

233 

233 

233 

05 

mailbox_test 

169 

174 

0 

procinfo 

225 

225 

0 

sho_db_users 

140 

141 

sho_queues 

46 

46 

46 

include 

8202 

7669 

10358 

HA  probes 

3100 

Server  Total 

344462 

319762 

212917 

Version  8  Components 

Version  8 

Version  7.5 

Version  7 

CAMINHelp 

9641 

C 

© 

HTML 

5662 

632 

pO 

68 

Servlets 

17339 

13243 

0 

■ 

Applets 

43435 

0 

a 

< 

Web  Total 

77173 

57310 

0 
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Version 8 Components  _ Versions  Version 7.5  Version? 


CW  Inventory  Reports 

450C 

4300 

CWPF  Reports 

1900 

1900 

pS 

Weapons  &  Agent  Reports 

IHHES 

900 

& 

s 

Installation  Reports 

700 

700 

u 

H 

Permission  Reports 

600 

600 

u 

o 

Request  Reports 

200 

200 

a 

Schedule  1  Reports 

1100 

1100 

1100 

Oracle  Reports 

2100 

0 

Reports  Total 

12000 

9700 

Version  8  Components 

Version  8 

Version  7.5 

Version  7 

Tables 

3410 

2674 

2561 

Views 

3294 

2010 

2010 

es  9 

indexes 

620 

510 

823 

«  2 
ts  ? 

grants 

422 

335 

335 

w  i 

e  X 

synonyms 

422 

328 

Backup  scripts 

2113 

Database  Total 

■■HI 

5857 

5729 

- — — — - - -  ■ 

E 

System  Totals 

831519 

745798 

574762 

APPENDIX  D.  CAMIN  CONTRACTOR  METRICS 


[From  Ref.  3] 


Requirements 

❖  Requirements  Identified 

❖  Requirements  fulfilled 

❖  Requirements  outstanding 

❖  Requirements  validated  (for  a  release) 

❖  Requirements  by  release 

*t*  Requirements  by  Configuration  Item 

❖  Requirements  by  category  (new,  baseline  change,  design 
change,  implementation  change) 

❖  Changes  per  configuration  item  (volatility) 

Program 

Management 

❖  Cost  to  complete 

❖  Hours  expended  by  day,  month,  task  order/project 

❖  Hours  to  complete 

♦♦♦  Cost  expended  by  month,  task  order/project 

❖  Baseline  (cost/hours/duration)  vs.  actual  &  progress  (% 
complete) 

❖  Software  size  by  version  (SLOC) 

❖  Software  size  by  tasking  (function  point) 

Quality  Assurance 

❖  Quality  Audits  planned/conducted 

❖  Nonconforming  conditions  found 
♦♦♦  Nonconforming  conditions  closed 

❖  Open  nonconforming  conditions. 

Configuration 

Management 

❖  Backup  success 

❖  Backup  resources  used 

❖  Hardware  Environment  Changes 

❖  Software  Environment  Changes 

❖  Number  of  files  checked  out 
♦t*  Number  of  files  changed 

♦♦♦  Number  of  users  who  changed  a  file 

❖  Number  of  files  changed 

❖  Number  of  times  each  file  was  checked  out 

❖  Number  of  users  who  checked  out  a  file 

❖  Number  of  users  who  changed  a  file 

❖  Number  of  lines  that  changed  per  file  (code  only) 

❖  Average  number  of  changed  lines  (code  only) 

♦♦♦  Average  length  of  time  a  file  was  checked  out 

Training 

❖  Training  Requirements,  vs.  training  fulfilled 

❖  Effort/hours  expended  in  staff  familiarization  (training) 
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Peer  Review 

Peer  Reviews  planned/conducted 

❖  Problems/Defects  identified 
♦♦♦  Problems/Defects  closed 

❖  Time  to  Closure 

Critical  Resoiirce 

❖  Staffing  Needed  by  Lifecycle 

❖  Equipment  resources  needed  by  Lifecycle 

Maintenance 

Problem  Calls  logged  (by  category,  by  users,  by  action) 
Time/effort  to  service  calls 

❖  Open  problem  calls 

❖  Data  changes/interventions  logged 

❖  Time/effort  to  make  data  change 

Development 

lime  to  implement  (baseline  vs.  actual)  by  software 
change/Configuration  Item 
♦J*  Defects/Problems  identified 
♦♦♦  Defects/Problems  corrected 
♦♦♦  Defects/Problems  validated 
❖  Time  to  closure  (date  corrected  -date  identified) 

♦♦♦  Effort  to  correct 

Intergroup 

Coordination 

❖  Action  Items  logged 
♦♦♦  Action  Items  open 

❖  Time  to  close  items 
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APPENDIX  E.  CAMIN  SCREEN  CAPTURES 


This  appendix  shows  samples  of  screen  captures  from  the  CAMIN  system.  The 
data  displayed  in  these  screens  is  from  the  “training”  database,  and  are  likely  to  be 
invented  data.  This  rudimentary  hierarchy  of  the  CAMIN  interface  helps  explain  how  the 
following  screens  fit  into  the  larger  CAMIN  system. 


CAMIN  Applications 

CAMIN  Web  Site 

•  General  Data 

•  Calendar 

o  Declared  Chemical  Information 

•  Report  Generator 

o  Chemical  Weapons  Items 

•  Notification  Archive 

o  Site  Information 

•  Site  Diagrams 

■  Installations  and  Plant  Sites  (Site 

•  Process  Flow  Diagrams 

Diagrams) 

•  Newsletters 

■  Declared  Facilities  and  Plants 

•  rrR  Info 

■  Building  Information  (Planographs) 

•  DMUG  Info 

•  Data  Access 

D  Alerts 

•  Software  Updates 

o  Data  Download 

•  Training  Information 

o  Site  Diagram  Modifications 

•  Policy  Information 

o  Reports  Viewer 

•  Documentation 

•  Stock  Records 

•  Help 

o  Stock  Records  Maintenance 

•  Administration 

o  Materiel  Release  Information 

•  Links 

o  Document  Register 

o  Annual  Inventory 

o  CW  Destruction  Reporting 

•  CWPF  History 

•  Schedule  1  Permitted  Activities 

•  Administrative  Tools/User  Administration 

•  Common  Tools 

o  CAMIN  Settings 

o  Data  Administration 

o  Ad  Hoc  Query 

•  Notifications  and  Declarations 

o  Notifications  Generation  and  Coordination 

G  Notifications  Catalog 

o  Declarations  Generation  and  Coordination 

o  Declarations  Catalog 
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Site  Information  >  Installations  and  Plant  Sites 


fie  £<&  j2atd  geiwellrrfo  fieports  QA  )£/hckm  Help 


]tlH|#|  '^)  H I  < }  ►  I M I fj 


PI 

i 

. '  '  •• 

IS 

Is 

m  'iv- 

11^1 

- 

Name:  |Blue  Grass  CW  Storage  Facftty 

.  .  Jll  Becord  f  1 

"r“ 

3  i 

type: 

f  jCw  Storage  Facility 

_  ;  Blue  Grass  Chemical  Activily  2091 

“3  ' 

Status: 

[Active 

jKingston  Highway 

.  — „ 

-4  : 

t^Stude: 

137/43/08N 

iRichmond 

ton^lude: 

j084/^13/00fV/ 

"3 

TSCode: 

ff  SC  Fac-6 

Zip:  [40475-5008  ^ 

Percent  Capaci^C^): 

1  mSS 

■  [united  Stales  of  America 

•3 

Instal^m  jBlue  Grass  Chemical  Activity 

Owner  )mAJ  JohnM.  Riley 

^  :  of  Last  Inspedion:  {Routine  CW  Storage 

Operator:  jMAJ  JohnM.  Riley 

_ _ 

|n$p.lnfa..  [ 

-  A . 

Epetrrfo... 

Name 

. . . .  1 

Offe®  1  Function  f.  OfTkie  Phone  l^j 

1 

Deborah  L  Boston 

AMSSB'OBG'TO  Treaty  Officer  859-825-6285 

■  ■ 

— 

....: - 

.NUM  ; 


The  screen  shown  here  displays  the  types  of  data  CAMIN  stores  for  an 
installation.  CAMIN  also  has  a  screen  for  each  facility  that  resides  within  an  installation 


or  plant  site.  CAMIN  workstation  users,  with  permission,  can  change  the  information. 
These  data  are  used  by  both  treaty  and  accoimtability  applications  in  CAMIN. 
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Site  Information  >  Building  Information  (for  Storage  facility) 


Site  Information  <  [Building  Information] 


:  ^  Fie  £cR  Data  Gen«^lnfo  Reports  2A  View  V^b-kJow  Hdp  ;• 


^2Si\ 


13  of  ;| 

PBCA,  Bond  Road  Exclusion  Area 

J 

Btidinglnfo... 


.  irePiSE 

im  £2-170 


tructtffe  I  ID 


62-170  Undergoing  Annual  inventorv  None  None  Yes 


62-180 

62-180 

;  Ad 

62-200 

^  62-200 

■  Ad 

62-210 

62-210 

Ad 

62-220 

i  62-220 

;  Ad 

62-230 

■  62-230 

:  Acl 

62-240 

62-240 

Ac! 

62-250 

62-250 

Acl 

62-260 

62-260 

1  Acl 

62-270 

62-270 

Act 

62-280 

62-280 

Act 

62-290 

62-290 

Act 

62-300 

;  62-300 

62-310 

62-310 

Acti 

62-320 

Acti 

62-330 

62-330 

Acti 

62-:wb 

62-34b 

Acti 

62-370 

62-370 

Acti 

62-380 

;  62-380 

Acti 

62-400 

62-400 

Actr 

;U.P0AT|.;  lOPENINONE  .  JNONE  i  jNUM  j  A 


This  screen  shows  an  example  of  a  building  list  in  CAMIN  for  a  storage  facility. 
Each  building  may  have  unique  grid  layout  that  users  would  establish  within  this 
application.  The  first  building  on  the  list  is  currently  imdergoing  annual  inventory.  The 
Annual  Inventory  application  in  CAMIN  allows  scheduling,  printing  out  inventory 
worksheets,  entering  the  completed  inventory,  and  notifying  the  accountable  officer  of 
completion. 


Site  Information  >  Building  Information  (for  CWPF  facility) 


This  screen  shows  a  listing  of  buildings  that  would  be  associated  with  a  former 
CW  Production  Facility  (CWPF).  The  same  application  is  used  as  for  the  last  screen,  but 
for  a  different  purpose.  Through  reuse,  CAMIN  has  saved  development  costs  and 
avoided  duplicate  maintenance  costs. 
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Planograph 


The  Planograph  is  a  map  of  a  storage  structure,  showing  grids  where  stock  is 
stored.  Each  grid  shown  on  the  screen  is  listed  on  the  next  page,  with  the  contents  of  that 
grid.  Users  enter  the  data  used  in  creating  the  Planograph  through  the  Stock  Records 
Maintenance  application. 
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This  screen  represents  the  second  page  of  a  Planograph.  This  listing  shows  the 
contents  of  the  structure  by  grid  location.  The  Planograph  is  beneficial  in  annual 


inventories,  maintenance  activities,  emergency  response,  and  in  treaty  inspections. 
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Chemical  Weapons  Item  Information 


^bbI, 


Info ; :  Repoi 

ts  SA  View  Window 

14  1  4  1  1  M  1  9 1  1 

Commwi  Name  of  ;Ch«i^  W  IUnknown 


Type:  .  p 

Dasi^on  ^  f 

Size/CaSben  f 

FiWeight  f 

F®  Volume  [cam^  f* 

Deslfuctiai  Categc^  p 

treafyCafegoiy:  p 

Nomend^uer  p 

Declared  under  CV^  and  BDA: 
BultCdr^aiien  n 


jBomb 

jM47[SusH)  ^  ^ 

pOOlb  _ 

I  27.22000  .^ 
f"  ;o33bocidoo 
[category  1  (Schedule  1  Fill] 
[Munition 

|UNK  Bomb  M47(SusH)  1001b 


MARB  Nurribers 


MARS  mfa,. " 


Rea»  .  .  ;  >8.03  Hrem  Wf^ATE  ;0PEM  jMETRIC  jKlogiams  {  ^ 


This  application  provides  a  dictionary  of  all  CW  items  in  the  CAMIN  system. 
The  application  creates  a  structured  format  for  all  nomenclatures,  in  accordance  with  the 
Chemical  Weapons  Convention.  The  application  also  creates  a  cross  reference  between 
nomenclatures  and  National  Stock  Numbers,  to  conform  to  the  Army  cataloging  system. 
The  nomenclature  consists  of  four  elements:  1)  the  name  of  the  chemical  fill,  2)  the  type 
of  item,  3)  the  military  designator,  and  4)  the  size  or  caliber  of  the  item.  “None“  or 
“unknown”  are  acceptable  for  almost  every  field. 
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Declared  Chemical  Information 


ge  £dit  Data  General  Info  fiepoHs  C|A  yiew  jij/hdow  Hefc 


Record 


Gonmori  Name:  ; 

lUPAC  Water  and  Bleach 

Npnr^ndature: 


Designator  Water  and  Bleach 


Schecye: 


I  Unscheduled 


CAS  Registry  #:  iNone-WB 


IS  Code: 


Sp^cSic  Gf^ily: 


|TSCChem-50 


1.23457 


Toxid^ 

Princ^^ 

Re^tion 

Product 


J  very  toxic 


j^ain  JUPDATE  pDPEN  jNONE  [TJ 


Each  chemical  fill  in  CAMIN  is  defined  in  this  application.  The  chemical  must 
be  defined  before  a  CW  Item  can  contain  that  fill. 

Both  the  CW  Items  and  the  Declared  Chemicals  Applications  perform  as 
dictionaries  for  CAMIN.  Any  change  to  these  screens  may  affect  multiple  items  located 
at  multiple  sites. 
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Stock  Record  Card  for  Chemical  Items 


ffe  General  Info  Beports  fiA  5^«dow  He^P 


BlHlifel  ll^liii H H  !►  I H \  f i  1 


'l'i"i'^,«.^ 


Imlalaiion:  jbeseret  ChemicalJDeiM^ 

Dedared  Fsc^ ,  jDesetet  CW  Staage  Fadtty 


3 


Record 


1  of 


2 


Nomenctehie:  jGB  OneJa^Ca^rT^c  D  ^ 

SlockNunnber  |l  3^2939239 


Lol  Number  jRM76033-:K7 

jon' I  ■  /  iTia 


13 


Transachon 

Dale 


Transadion 

Slabus 


Custocial 

IdCatioh 


Treaty 

location 


SeriairrSidlOP] 


cc 


Stock 


Amount 

Agent 


1103  1103  D-84180  AFAB  I  A  1^ 


Die 


Docun«rA 

rNumber 


5081 


I  i:i?-Apr-2Gi:n  ATR  Pending -oar  eg  1103  TOCDF  D-84160  AFAE 


OeSO;:  AIR  M  TOCDF-01 097-1 004 


■LmJ— 


Stock  Baiance:  | 
AgentBalance:  f 

Hfiz.  frrfo. . 


Rem^ks; 


IA1103P9726812 


.880388987 


3 


1  Reack> 

attain 

;UroATE  >OPEN  iMETRIC  ;Tonnes  | 

;NUM  r“ 

_ s 

This  screen  displays  a  single  lot  of  an  item,  stored  at  a  single  facility.  From  this 
screen,  the  user  with  appropriate  permissions  at  the  facility  can  change  characteristics  of 
items  or  move  items  to  another  grid,  building,  facility,  or  installation. 
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Document  Associated  with  Transaction 


The  CAMIN  creates  documents  that  are  required  by  Army  regulation  for 
management  of  wholesale  Army  property.  Regulations  require  these  documents  for 
changes  to  status  or  shipments. 
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Data  Download 


This  application  was  not  an  original  system  requirement.  It  was  part  of  a  project 
to  compensate  for  slow  network  connections  at  the  Johnston  Island  facility.  The  Data 
Download  application  populates  the  lookup  tables  that  reside  on  CAMIN  workstations. 
Lookup  tables  contain  data  from  the  central  server  that  changes  infrequently.  The 
CAMIN  software  uses  the  lookup  tables  to  populate  screens  without  receiving  all  data 
from  the  central  server. 
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■BSSSH 

KiHIWPfg 

Current  CAf^IN  Version:  8.03 


HOME  ^CALENDAR  ;  DATA  ACCESS  f  SYSTEM  INFO  SOFTWARE  UPDATES  i  TRAINING  ;  HELP  i  ADMIN 


CAMlNHeip  Line:  420~436-3iS9 
CAMm  Pager:  1-$00~SKY~8888  1714303) 

Semi  cfTtaii  to  tfte  P-sger  ^  i  I 


•  Home 

•  Calendar 
•Report  Generator 

•  Notification  Archive 
•Site  Diagrams 
•Process  Flow  Diagrams 
•Newsletters 

•CCB  Info 
•DMUG  Info 

•  Software  Updates 

•  Training  Information 

•  Policy  Information 

•  Documentation 
•Help 

•  Administration 

Links 

SBCCOy/  Home  Page 
U.S.  Army  Home  Page 
OPCIV  Home  Page 
PNCD  Home  Page 
PNCD  CWC  Net 


Announcements 

No  current  anruyuncements 


•  ^  i 


Meetings 

Training; 

No  oirrent  announcenoonts 

•  CAMIN  Classroom  Training  for 

CWPF  and  Schedule  l  17  -  19  April. 

C^ehdar  1  HMeebmi  Archive  1 

iMorelrifo  i 

To  provide  feedback  to  the  webmaster,  click  here. 


-  i;  it®  Irtemet 


The  CAMIN  Web  Site  provides  access  to  commonly  needed  data  from  the 


CAMIN  server.  The  data  presented  by  this  web  site  is  from  the  same  source  as  the  client 
system.  The  web  site  users  can  also  get  programmatic  information,  training,  help,  and 
calendar  information. 


Ultimately,  this  web  site  will  support  an  increasing  portion  of  the  CAMIN  client 
(workstation)  capability.  In  this  way,  the  PM  can  reduce  the  dependence  on  workstations 
and  increase  system  access. 
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